Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 66514 invoked from network); 22 Apr 2009 15:17:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 22 Apr 2009 15:17:29 -0000 Received: (qmail 40848 invoked by uid 500); 22 Apr 2009 15:17:25 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 40797 invoked by uid 500); 22 Apr 2009 15:17:25 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 40767 invoked by uid 99); 22 Apr 2009 15:17:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Apr 2009 15:17:25 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of tamirkamara@gmail.com designates 74.125.78.27 as permitted sender) Received: from [74.125.78.27] (HELO ey-out-2122.google.com) (74.125.78.27) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Apr 2009 15:17:15 +0000 Received: by ey-out-2122.google.com with SMTP id d26so8969eyd.35 for ; Wed, 22 Apr 2009 08:16:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=JZ78+I9Mbv7fl3eNWKd9BdDbM81SF8BIidFFJmenkJc=; b=kzEp0Xf3q/GtGs7UVBeF9arfbjLkgJI/sAmuvLGs1SYqRInhgbDDDjLo//Kkah0Fk6 QFdZanuJklJxpstI7hdxDosu0iwhCYFMmWQwIAmrcsaC/T6GptJnqwTL17V7ZOc5vuxd ztgWZZCgLPhVIxgs2fJYeFS8nLSwWp3GGjoJ4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=vZgpK4OGfT0xMITQMmPD9Mfwn/0MeVJmeSblpNf3Oq3zl4qnofG5rF/YX82FLgiNi8 Wx8olBXx2gAokktjUhpHaAaDKzKAgoCEICicGt9+1fSsJaQ7p4UHtD0HtwR8/fJNn/nz tCJyN1KEzkR/2Za5OK7TBVvSazR4+AsxLYXks= MIME-Version: 1.0 Received: by 10.216.1.202 with SMTP id 52mr718462wed.15.1240413414766; Wed, 22 Apr 2009 08:16:54 -0700 (PDT) Date: Wed, 22 Apr 2009 18:16:54 +0300 Message-ID: <6d10e930904220816r16cf8246qb8f43604a1eb5d7b@mail.gmail.com> Subject: NameNode Startup Problem From: Tamir Kamara To: core-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00163646db76f63a6a046826400f X-Virus-Checked: Checked by ClamAV on apache.org --00163646db76f63a6a046826400f Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi, After a while working with hadoop I'm now faced with a situation where the namenode won't start up. I'm working with a patched up version of 0.19.1 with ganglia patches (3422, 4675) and with 5269 which suppose to deal with killed_unclean task status and the massive "serious problem" lines in the JT logs. The latest NN logs are below. Can you help me figure out what is going on ? Thanks, Tamir 2009-04-22 18:12:36,966 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = lb-emu-3/192.168.14.11 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.19.2-dev STARTUP_MSG: build = -r ; compiled by 'tkamara' on Tue Apr 21 12:03:50 IDT 2009 ************************************************************/ 2009-04-22 18:12:37,448 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310 2009-04-22 18:12:37,456 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: lb-emu-3.israel.verisign.com/192.168.14.11:54310 2009-04-22 18:12:37,467 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2009-04-22 18:12:37,474 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullC ontext 2009-04-22 18:12:37,627 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop,hadoop 2009-04-22 18:12:37,628 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2009-04-22 18:12:37,628 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2009-04-22 18:12:37,649 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.sp i.NullContext 2009-04-22 18:12:37,651 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2009-04-22 18:12:37,814 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 3427 2009-04-22 18:12:38,486 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 28 2009-04-22 18:12:38,511 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 488333 loaded in 0 seconds. 2009-04-22 18:12:38,634 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /usr/local/hadoop-datastore/hadoop/dfs/name/current/edits of size 82110 edits # 477 loaded in 0 seconds. 2009-04-22 18:12:40,893 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Invalid opcode, reached end of edit log Number of transactions found 36635 2009-04-22 18:12:40,893 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /usr/local/hadoop-datastore/hadoop/dfs/name/current/edits.new of size 5229334 edits # 36635 l oaded in 2 seconds. 2009-04-22 18:12:41,024 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: saveLeases found path /tmp/temp623789763/tmp659456056/_temporary/_attempt_200904211331_0010_r_000002_0/part-00002 but no matching entry in namespace. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4608) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1010) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1031) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:288) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:208) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:194) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868) 2009-04-22 18:12:41,038 INFO org.apache.hadoop.ipc.Server: Stopping server on 54310 2009-04-22 18:12:41,038 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: saveLeases found path /tmp/temp623789763/tmp659456056/_temporary/_attempt_20090 4211331_0010_r_000002_0/part-00002 but no matching entry in namespace. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4608) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1010) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:1031) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:288) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:208) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:194) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868) 2009-04-22 18:12:41,039 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at lb-emu-3/192.168.14.11 ************************************************************/ --00163646db76f63a6a046826400f--