hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-1864) Hadoop Namenode not starting up.
Date Wed, 27 Apr 2011 04:41:03 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron T. Myers resolved HDFS-1864.
----------------------------------

    Resolution: Invalid

Hey Ronak, this isn't really something that should be tracked in JIRA, since we can't identify
what the problem is. Apache JIRA is to track project development, not troubleshooting. Instead,
you should try emailing a *-user@ list with the description of your issue. Since, judging
by the version strings in the stack trace, you're running CDH3b3, I'd recommend emailing the
cdh-user@cloudera.org mailing list.

Best,
Aaron

> Hadoop Namenode not starting up.
> --------------------------------
>
>                 Key: HDFS-1864
>                 URL: https://issues.apache.org/jira/browse/HDFS-1864
>             Project: Hadoop HDFS
>          Issue Type: Task
>            Reporter: Ronak Shah
>
> 1. Checked to make sure hadoop was running properly. Discovered that we suppose to run
'jps' and make sure there is a namenode process. 
> 2. Documentation said, if namenode does not exist - then run 
> /etc/init.d/hadoop-0.20-namenode start 
> /etc/init.d/hadoop-0.20-namenode status - namenode process fails 
>  EQX hdfs@hadoop-master:/usr/lib/hadoop/bin$ /etc/init.d/hadoop-0.20-namenode status

> namenode dead but pid file exists 
> 3. Searched for pid files. We deleted pid files. 
> 4. All over stats fell off. As direct of result, looking at the process list - and there
appeared to be a stalled process that was killed. 
> kill -9 
> for the following process: 
> EQX root@hadoop-master:/etc/init.d# ps aux | grep namenode 
> hdfs 5038 0.2 1.0 3617440 526704 ? Sl Mar31 74:02 /usr/java/default/bin/java -Dproc_namenode
-Xmx3000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dhadoop.log.dir=/usr/lib/hadoop/logs
-Dhadoop.log.file=hadoop-hdfs-namenode-hadoop-master.rockyou.com.log -Dhadoop.home.dir=/usr/lib/hadoop
-Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,DRFA -Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64
-Dhadoop.policy.file=hadoop-policy.xml -classpath /usr/lib/hadoop/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop:/usr/lib/hadoop/hadoop-core-0.20.2+737.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/hadoop-fairscheduler-0.20.2+737.jar:/usr/lib/hadoop/lib/hadoop-lzo-0.4.8.jar:/usr/lib/hadoop/lib/hadoop-lzo.jar:/usr/lib/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop/lib/hue-plugins-1.1.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-6.1.14.jar:/usr/lib/hadoop/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop/lib/junit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.2.2.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-api-2.1.jar::/usr/local/lib/mysql-connector-java-5.1.7-bin.jar:/usr/local/lib/mail.jar:/usr/local/lib/mysql-connector-java-5.1.7-bin.jar:/usr/local/lib/mail.jar:/usr/local/lib/mysql-connector-java-5.1.7-bin.jar:/usr/local/lib/mail.jar:/usr/local/lib/mysql-connector-java-5.1.7-bin.jar:/usr/local/lib/mail.jar:/usr/local/lib/mysql-connector-java-5.1.7-bin.jar:/usr/local/lib/mail.jar
org.apache.hadoop.hdfs.server.namenode.NameNode 
> root 16449 0.0 0.0 61136 744 pts/4 S+ 16:29 0:00 grep namenode 
> EQX root@hadoop-master:/etc/init.d# kill -9 5038
>  We starting looking at log output - we discovered the namenode startup process is throwing
a null pointer exception. 
> STARTUP_MSG: build = -r 98c55c28258aa6f42250569bd7fa431ac657bdbd; compiled by 'root'
on Mon Oct 11 13:14:05 EDT 2010 
> ************************************************************/ 
> 2011-04-25 21:16:47,841 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=NameNode, sessionId=null 
> 2011-04-25 21:16:47,949 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.ganglia.GangliaContext31

> 2011-04-25 21:16:47,982 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hdfs

> 2011-04-25 21:16:47,982 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=root

> 2011-04-25 21:16:47,982 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true

> 2011-04-25 21:16:47,987 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 
> 2011-04-25 21:16:48,301 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.ganglia.GangliaContext31

> 2011-04-25 21:16:48,302 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean 
> 2011-04-25 21:16:48,328 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of
files = 237791 
> 2011-04-25 21:16:51,699 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of
files under construction = 0 
> 2011-04-25 21:16:51,699 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file
of size 42758182 loaded in 3 seconds. 
> 2011-04-25 21:16:51,701 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException

> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1088)

> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1100)

> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:987)

> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:974)

> at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:718) 
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1034) 
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:845) 
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:379)

> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:99)

> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:343)

> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:317)

> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:214) 
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:394) 
> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1148)

> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1157) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message