Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of thealy@bnl.gov designates
 130.199.3.132 as permitted sender)
Message-ID: <4FA3DFFA.9080507@bnl.gov>
Date: Fri, 04 May 2012 09:56:10 -0400
From: Terry Healy <thealy@bnl.gov>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:12.0) Gecko/20120430 Thunderbird/12.0.1
MIME-Version: 1.0
To: HDFS USER mail list <hdfs-user@hadoop.apache.org>
Subject: Unable to restart 1.0.2 NN, DFS
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Running Apache 1.0.2 Primary NN on Ubuntu 11.10, 8 datanodes. When I
came in today I ran a stop-all.sh since one of the datanodes was not
showing on the dfsnodelist.jsp?whatNodes=LIVE status page. I then ran
start-all.sh, and the NN dies on a Null Pointer Exception. I looked at
the source but cannot determine what is going. Below is the trace - any
explanation of cause or suggestions to get the system back up would be
appreciated.

** I have removed the prefix "org.apache.hadoop." from the list below in
the interest of saving space. **

-Terry

2012-05-04 09:45:02,046 INFO hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = xxxx/xxxxxxxx
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.0.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r
1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-05-04 09:45:02,229 INFO metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties
2012-05-04 09:45:02,244 INFO metrics2.impl.MetricsSourceAdapter: MBean
for source MetricsSystem,sub=Stats registered.
2012-05-04 09:45:02,246 INFO metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2012-05-04 09:45:02,246 INFO metrics2.impl.MetricsSystemImpl: NameNode
metrics system started
2012-05-04 09:45:02,483 INFO metrics2.impl.MetricsSourceAdapter: MBean
for source ugi registered.
2012-05-04 09:45:02,488 WARN metrics2.impl.MetricsSystemImpl: Source
name ugi already exists!
2012-05-04 09:45:02,494 INFO metrics2.impl.MetricsSourceAdapter: MBean
for source jvm registered.
2012-05-04 09:45:02,496 INFO metrics2.impl.MetricsSourceAdapter: MBean
for source NameNode registered.
2012-05-04 09:45:02,530 INFO hdfs.util.GSet: VM type       = 64-bit
2012-05-04 09:45:02,530 INFO hdfs.util.GSet: 2% max memory = 17.77875 MB
2012-05-04 09:45:02,530 INFO hdfs.util.GSet: capacity      = 2^21 =
2097152 entries
2012-05-04 09:45:02,530 INFO hdfs.util.GSet: recommended=2097152,
actual=2097152
2012-05-04 09:45:02,560 INFO hdfs.server.namenode.FSNamesystem:
fsOwner=thealy
2012-05-04 09:45:02,560 INFO hdfs.server.namenode.FSNamesystem:
supergroup=supergroup
2012-05-04 09:45:02,560 INFO hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2012-05-04 09:45:02,567 INFO hdfs.server.namenode.FSNamesystem:
dfs.block.invalidate.limit=100
2012-05-04 09:45:02,567 INFO hdfs.server.namenode.FSNamesystem:
isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
accessTokenLifetime=0 min(s)
2012-05-04 09:45:02,749 INFO hdfs.server.namenode.FSNamesystem:
Registered FSNamesystemStateMBean and NameNodeMXBean
2012-05-04 09:45:02,765 INFO hdfs.server.namenode.NameNode: Caching file
names occuring more than 10 times
2012-05-04 09:45:02,775 INFO hdfs.server.common.Storage: Number of files
= 11
2012-05-04 09:45:02,783 INFO hdfs.server.common.Storage: Number of files
under construction = 0
2012-05-04 09:45:02,783 INFO hdfs.server.common.Storage: Image file of
size 11087 loaded in 0 seconds.
2012-05-04 09:45:02,785 ERROR hdfs.server.namenode.NameNode:
java.lang.NullPointerException
	at hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1094)
	at hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1106)
	at hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:1009)
	at
hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:208)
	at hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:626)
	at hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1015)
	at hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:833)
	at hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:372)
	at hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
	at hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
	at hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
	at hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
	at hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
	at hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
	at hdfs.server.namenode.NameNode.main(NameNode.java:1288)