Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <47BD7CB1.5090806@andremartin.de>
Date: Thu, 21 Feb 2008 14:29:21 +0100
From: =?ISO-8859-1?Q?Andr=E9_Martin?= <mail@andremartin.de>
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Namenode fails to re-start after cluster shutdown
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Hi everyone,
I downloaded the nightly build (see below) yesterday and after the 
cluster worked fine for about 10 hours I got the following
error message from the DFS client even all data nodes were up:

> 08/02/21 14:04:35 INFO fs.DFSClient: Could not obtain block 
> blk_-400895070464649
> 0788 from any node:  java.io.IOException: No live nodes contain 
> current block

So I decided to restart our cluster but unfortunately the Namenode
fails to start with the following exception:

> 2008-02-21 14:20:48,831 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = se09/141.76.44.209
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 2008-02-19_11-01-48
> STARTUP_MSG:   build = 
> http://svn.apache.org/repos/asf/hadoop/core/trunk -r 628999; compiled 
> by 'hudson' on Tue Feb 19 11:09:05 UTC 2008
> ************************************************************/
> 2008-02-21 14:20:49,367 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
> Initializing RPC Metrics with serverName=NameNode, port=8000
> 2008-02-21 14:20:49,374 INFO org.apache.hadoop.dfs.NameNode: Namenode 
> up at: se09.inf.tu-dresden.de/141.76.44.209:8000
> 2008-02-21 14:20:49,378 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
> Initializing JVM Metrics with processName=NameNode, sessionId=null
> 2008-02-21 14:20:49,381 INFO org.apache.hadoop.dfs.NameNodeMetrics: 
> Initializing NameNodeMeterics using context 
> object:org.apache.hadoop.metrics.spi.NullContext
> 2008-02-21 14:20:49,501 INFO org.apache.hadoop.fs.FSNamesystem: 
> fsOwner=amartin,students
> 2008-02-21 14:20:49,501 INFO org.apache.hadoop.fs.FSNamesystem: 
> supergroup=supergroup
> 2008-02-21 14:20:49,501 INFO org.apache.hadoop.fs.FSNamesystem: 
> isPermissionEnabled=true
> 2008-02-21 14:20:49,788 INFO org.apache.hadoop.ipc.Server: Stopping 
> server on 8000
> 2008-02-21 14:20:49,790 ERROR org.apache.hadoop.dfs.NameNode: 
> java.io.IOException: Created 13 leases but found 4
>     at 
> org.apache.hadoop.dfs.FSImage.loadFilesUnderConstruction(FSImage.java:935)
>     at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:749)
>     at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:634)
>     at 
> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:223)
>     at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:79)
>     at 
> org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:261)
>     at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:242)
>     at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:131)
>     at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:176)
>     at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:162)
>     at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:851)
>     at org.apache.hadoop.dfs.NameNode.main(NameNode.java:860)
>
> 2008-02-21 14:20:49,791 INFO org.apache.hadoop.dfs.NameNode: 
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at se09/141.76.44.209
> ************************************************************/

Any ideas?

Cu on the 'net,
                        Bye - bye,

                                   <<<<< Andr� <<<< >>>> �rbnA >>>>>