hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamir Kamara <tamirkam...@gmail.com>
Subject NameNode Startup Problem
Date Wed, 22 Apr 2009 15:16:54 GMT
Hi,

After a while working with hadoop I'm now faced with a situation where the
namenode won't start up. I'm working with a patched up version of 0.19.1
with ganglia patches (3422, 4675) and with 5269 which suppose to deal with
killed_unclean task status and the massive "serious problem" lines in the JT
logs.
The latest NN logs are below.

Can you help me figure out what is going on ?

Thanks,
Tamir



2009-04-22 18:12:36,966 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = lb-emu-3/192.168.14.11
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.19.2-dev
STARTUP_MSG:   build =  -r ; compiled by 'tkamara' on Tue Apr 21 12:03:50
IDT 2009
************************************************************/
2009-04-22 18:12:37,448 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=54310
2009-04-22 18:12:37,456 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
lb-emu-3.israel.verisign.com/192.168.14.11:54310
2009-04-22 18:12:37,467 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2009-04-22 18:12:37,474 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing
NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullC
ontext
2009-04-22 18:12:37,627 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop,hadoop
2009-04-22 18:12:37,628 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2009-04-22 18:12:37,628 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2009-04-22 18:12:37,649 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.sp
i.NullContext
2009-04-22 18:12:37,651 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2009-04-22 18:12:37,814 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 3427
2009-04-22 18:12:38,486 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 28
2009-04-22 18:12:38,511 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 488333 loaded in 0 seconds.
2009-04-22 18:12:38,634 INFO org.apache.hadoop.hdfs.server.common.Storage:
Edits file /usr/local/hadoop-datastore/hadoop/dfs/name/current/edits of size
82110 edits # 477 loaded in
 0 seconds.
2009-04-22 18:12:40,893 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Invalid opcode, reached
end of edit log Number of transactions found 36635
2009-04-22 18:12:40,893 INFO org.apache.hadoop.hdfs.server.common.Storage:
Edits file /usr/local/hadoop-datastore/hadoop/dfs/name/current/edits.new of
size 5229334 edits # 36635 l
oaded in 2 seconds.
2009-04-22 18:12:41,024 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.io.IOException: saveLeases found path
/tmp/temp623789763/tmp659456056/_temporary/_attempt_200904211331_0010_r_000002_0/part-00002
but no matching entry in namespace.
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4608)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1010)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1031)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:288)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
2009-04-22 18:12:41,038 INFO org.apache.hadoop.ipc.Server: Stopping server
on 54310
2009-04-22 18:12:41,038 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
saveLeases found path
/tmp/temp623789763/tmp659456056/_temporary/_attempt_20090
4211331_0010_r_000002_0/part-00002 but no matching entry in namespace.
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4608)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1010)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1031)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:288)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)

2009-04-22 18:12:41,039 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at lb-emu-3/192.168.14.11
************************************************************/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message