hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Gammeter <gamme...@vision.ee.ethz.ch>
Subject Re: While starting HDFS process getting stucked.
Date Fri, 16 Sep 2011 10:27:02 GMT
Try clearing your hadoop directories and reformatting the namenode, it seemed to help in this
case: (cf 
http://web.archiveorange.com/archive/v/GJ8pzKvfDoYHyDQpVRSS ).

On 09/16/2011 12:21 PM, kiranprasad wrote:
> I am getting this below mentioned error when I tried to start the .sh files
>
> LOG:
> =====
>
> 2011-09-16 19:51:50,310 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = pig4/127.0.0.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
-r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2011-09-16 19:51:51,170 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC
Metrics with hostName=NameNode, port=8020
> 2011-09-16 19:51:51,197 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode
up at: 10.0.0.61/10.0.0.61:8020
> 2011-09-16 19:51:51,201 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=NameNode, sessionId=null
> 2011-09-16 19:51:51,203 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2011-09-16 19:51:51,474 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=kiranprasad.g,kiranprasad.g
> 2011-09-16 19:51:51,474 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
> 2011-09-16 19:51:51,474 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
> 2011-09-16 19:51:51,509 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2011-09-16 19:51:51,512 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
> 2011-09-16 19:51:52,355 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of
storage directory /tmp/hadoop-kiranprasad.g/dfs/name. Reported: -19.
> Expecting = -18.
> at org.apache.hadoop.hdfs.server.common.Storage.getFields(Storage.java:647)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.getFields(FSImage.java:542)
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.read(Storage.java:227)
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.read(Storage.java:216)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:301)
> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
> 2011-09-16 19:51:52,357 INFO org.apache.hadoop.ipc.Server: Stopping server on 8020
> 2011-09-16 19:51:52,573 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
Unexpected
> version of storage directory /tmp/hadoop-kiranprasad.g/dfs/name. Reported: -19. Expecting
= -18.
> at org.apache.hadoop.hdfs.server.common.Storage.getFields(Storage.java:647)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.getFields(FSImage.java:542)
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.read(Storage.java:227)
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.read(Storage.java:216)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:301)
> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> 2011-09-16 19:51:52,593 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at pig4/127.0.0.1
> ************************************************************/
>
> Regards
>
> Kiran.G
>
>
>
> -----Original Message----- From: Stephan Gammeter
> Sent: Friday, September 16, 2011 2:35 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: While starting HDFS process getting stucked.
>
> Are your HDFS nodes running? Did they complete the startup? What do the logs say?
>
> On machines where /dev/random ist starved (machines with not much load and maybe VMs)
i think there can be an issue with jetty (internal http server) that
> blocks during startup, because it wants to initialize the secure random number generator.
>
> if you see in your datanode logs, that they get stuck upon startup:
>
> stephaga@googolplex:/home/awesome/hadoop/hadoop $ head -n 30 logs/hadoop-awesome-datanode-bender15.log.2011-09-07
> 2011-09-07 16:47:11,712 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = bender15##################
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2-append
> STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
-r 1057313; compiled by 'awesome' on Fri Feb 18 15:36:52 CET 2011
> ************************************************************/
> 2011-09-07 16:47:19,051 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
FSDatasetStatusMBean
> 2011-09-07 16:47:19,054 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened
info server at 50010
> 2011-09-07 16:47:19,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing
bandwith is 16777216 bytes/s
> 2011-09-07 16:47:19,118 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
> 2011-09-07 16:47:19,191 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort()
before open() is -1. Opening the
> listener on 50075
> 2011-09-07 16:47:19,191 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort()
returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> 2011-09-07 16:47:19,191 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
> 2011-09-07 16:47:19,191 INFO org.mortbay.log: jetty-6.1.14
> ----> STUCK HERE
>
> then try adding the following line to your "hadoop-env.sh" :
>
> # cf: http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup
> # cf: http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom
> export HADOOP_OPTS="-Djava.security.egd=file:/dev/./urandom"
> #
>
> hope it helps,
>
> best,
> Stephan
>
> On 09/16/2011 10:54 AM, kiranprasad wrote:
>> Yes I ve formatted the namenode.
>> *From:* Sudharsan Sampath <mailto:sudhan65@gmail.com>
>> *Sent:* Friday, September 16, 2011 2:11 PM
>> *To:* hdfs-user@hadoop.apache.org <mailto:hdfs-user@hadoop.apache.org>
>> *Subject:* Re: While starting HDFS process getting stucked.
>> Have u formatted ur namenode ?
>> Thanks
>> Sudhan S
>>
>> On Fri, Sep 16, 2011 at 11:01 AM, kiranprasad <kiranprasad.g@imimobile.com <mailto:kiranprasad.g@imimobile.com>>
wrote:
>>
>> Hi
>>
>> I am new to Hadoop and PIG,
>>
>> For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 - Slaves)
>>
>> I ve installed PIG in 10.0.0.61 VM.=20
>>
>> Hadoop version : hadoop-0.20.2 and PIG : pig-0.8.1
>> I ve updated the xmls , please find the below
>>
>> mapred site.xml
>> --------------
>> <configuration>
>> <property>
>> <name>mapred.job.tracker</**name>
>> <value>10.0.0.61:8021 <http://10.0.0.61:8021></value>
>> </property>
>> </configuration>
>>
>>
>> core-site.xml
>> ----------
>> <configuration>
>> <property>
>> <name>fs.default.name <http://fs.default.name></name>
>> <value>hdfs://10.0.0.61:8020 <http://10.0.0.61:8020></**value>
>>
>> </property>
>> </configuration>
>>
>> Hdfs-site.xml
>> ----------------
>> <configuration>
>> <property>
>> <name>fs.default.name <http://fs.default.name></name>
>> <value>hdfs://10.0.0.61:8020 <http://10.0.0.61:8020></**value>
>> </property>
>> <property>
>> <name>mapred.job.tracker</**name>
>> <value>10.0.0.61:8021 <http://10.0.0.61:8021></value>
>>
>> </property>
>> <property>
>> <name>dfs.replication</name>
>> <value>2</value>
>> </property>
>>
>> masters
>> ---------
>> 10.0.0.61
>>
>> slaves
>> --------
>>
>> 10.0.0.62
>> 10.0.0.63
>>
>>
>> I ve tried with hadoop fs -ls but still facing the same problem.
>>
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/start-all.sh
>> starting namenode, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-nameno=
>> de-pig4.out
>> 10.0.0.62 <http://10.0.0.62>: starting datanode, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-datano=
>> de-pig3.out
>> 10.0.0.63 <http://10.0.0.63>: starting datanode, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-datano=
>> de-pig2.out
>> 10.0.0.61 <http://10.0.0.61>: starting secondarynamenode, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-second=
>> arynamenode-pig4.out
>> starting jobtracker, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-jobtra=
>> cker-pig4.out
>> 10.0.0.63 <http://10.0.0.63>: starting tasktracker, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-tasktr=
>> acker-pig2.out
>> 10.0.0.62 <http://10.0.0.62>: starting tasktracker, logging to
>>
>> /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-tasktr=
>> acker-pig3.out
>> [kiranprasad.g@pig4 hadoop-0.20.2]$
>> [kiranprasad.g@pig4 hadoop-0.20.2]$
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>>
>> After this it stopped running, it got stucked.
>>
>> Regards
>> Kiran.G
>>


Mime
View raw message