hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: HADOOP UPGRADE ISSUE
Date Thu, 22 Nov 2012 13:14:56 GMT
Make sure /opt/hadoop_newdata_dirr is 755 and not 777, and that it is owned
by the user that runs the DataNode (seems to be 'yogesh'). Once your DNs
are up, MR will fix itself.


On Thu, Nov 22, 2012 at 4:05 PM, yogesh dhari <yogeshdhari@live.com> wrote:

>  Hey Harsh,
>
> I have took down the instance on NN,
>
> and I started start-all.sh, but it doesn't run all demons.
>
> 1) Log file of DN...
>
> 2012-11-22 15:48:41,817 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
> dfs.data.dir: Incorrect permission for /opt/hadoop_newdata_dirr, expected:
> rwxr-xr-x, while actual: rwxrwxrwx
> 2012-11-22 15:48:41,817 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
> dfs.data.dir are invalid.
> 2012-11-22 15:48:41,817 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
> 2012-11-22 15:48:41,830 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>
> 2) Log file of TT...
>
> ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker
> because java.io.IOException: Call to localhost/127.0.0.1:9001 failed on
> local exception: java.io.IOException: Connection reset by peer
>     at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1075)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>     at org.apache.hadoop.mapred.$Proxy5.getProtocolVersion(Unknown Source)
>
> 3) Log file of NN...
> .
> .
> .
> .
> 2012-11-22 15:55:52,101 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:yogesh cause:java.io.IOException: File
> /opt/hadoop-0.20.2/hadoop_temporary_dirr/mapred/system/jobtracker.infocould only be replicated
to 0 nodes, instead of 1
> 2012-11-22 15:55:52,102 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call
> addBlock(/opt/hadoop-0.20.2/hadoop_temporary_dirr/mapred/system/
> jobtracker.info, DFSClient_-971904437, null) from 127.0.0.1:54047: error:
> java.io.IOException: File
> /opt/hadoop-0.20.2/hadoop_temporary_dirr/mapred/system/jobtracker.infocould only be replicated
to 0 nodes, instead of 1
> java.io.IOException: File
> /opt/hadoop-0.20.2/hadoop_temporary_dirr/mapred/system/jobtracker.infocould only be replicated
to 0 nodes, instead of 1
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
>     at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:601)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
> 2012-11-22 15:55:56,291 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
>
>
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Terminal's Output
>
> yogesh@yogesh-Aspire-5738:/opt/hadoop-1.0.4$ start-all.sh
> starting namenode, logging to
> /opt/hadoop-1.0.4/libexec/../logs/hadoop-yogesh-namenode-yogesh-Aspire-5738.out
> yogesh@localhost's password:
> localhost: starting datanode, logging to
> /opt/hadoop-1.0.4/libexec/../logs/hadoop-yogesh-datanode-yogesh-Aspire-5738.out
> yogesh@localhost's password:
> localhost: starting secondarynamenode, logging to
> /opt/hadoop-1.0.4/libexec/../logs/hadoop-yogesh-secondarynamenode-yogesh-Aspire-5738.out
> starting jobtracker, logging to
> /opt/hadoop-1.0.4/libexec/../logs/hadoop-yogesh-jobtracker-yogesh-Aspire-5738.out
> yogesh@localhost's password:
> localhost: starting tasktracker, logging to
> /opt/hadoop-1.0.4/libexec/../logs/hadoop-yogesh-tasktracker-yogesh-Aspire-5738.out
>
>
> yogesh@yogesh-Aspire-5738:/opt/hadoop-1.0.4$ jps
> 23297 JobTracker
> 23613 Jps
> 23562 TaskTracker
> 22658 NameNode
> 23205 SecondaryNameNode
>
>
>
> yogesh@yogesh-Aspire-5738:/opt/hadoop-1.0.4$ stop-all.sh
> stopping jobtracker
> yogesh@localhost's password:
> localhost: no tasktracker to stop
> stopping namenode
> yogesh@localhost's password:
> localhost: no datanode to stop
> yogesh@localhost's password:
> localhost: stopping secondarynamenode
>
>
>
>
> Please suggest
>
> Thanks & Regards
> Yogesh Kumar
>
>
>
>
> ------------------------------
> From: harsh@cloudera.com
> Date: Thu, 22 Nov 2012 15:18:14 +0530
>
> Subject: Re: HADOOP UPGRADE ISSUE
> To: user@hadoop.apache.org
>
> Ah alright, yes you can take down the shell instance of NN you've started
> by issuing a simple Ctrl+C, and then start it with the other DNs, SNN in
> the background with the regular start-dfs or start-all command. That is
> safe to do.
>
> P.s. Be sure to finalize your upgrade after adequate testing has been
> done. You can do this at a later time.
>
>
> On Thu, Nov 22, 2012 at 3:00 PM, yogesh dhari <yogeshdhari@live.com>wrote:
>
>  Hello Harsh..
>
> Thanks for your suggestion :-), I need more guidance over it.
>
> Terminal through I have run command
> hadoop namenode -upgrade (Say Terminal-1)
>
> terminal-1 got stuck at this point.
>
>
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 9 on 9000: starting
> ""and still cursor is blinking...""
>
> Should I do stop it using ctrl+c command.  If I do so it will kill the
> process.
>
> If I do open new terminal(say Terminal-2) and run JPS it shows
> 6374 NameNode
> 7615 Jps
>
> As you mentioned to run hadoop fs -ls /
> it shows all stored directories..( on Terminal-2 )
>
> Now. Should I kill the process over Termial-1. What should I do to make it
> complete without any loss of data..
>
> I am attaching screen shot..
>
> Please have a look
>
>
> Thanks & Regards
> Yogesh Kumar
>
>
>
>
>
>
>
>
> ------------------------------
> From: harsh@cloudera.com
> Date: Thu, 22 Nov 2012 14:14:00 +0530
> Subject: Re: HADOOP UPGRADE ISSUE
> To: user@hadoop.apache.org
>
>
> If your UI is already up, then NN is already functional. The UI merely
> shows that your upgrade is done but has not been manually finalized by you
> (leaving it open for a rollback if needed).
>
> You could try a simple "hadoop fs -ls /" to see if NN is functional, run
> some other regular job based tests of yours, and then finalize the new
> format by issuing "hadoop dfsadmin -finalizeUpgrade" to make the upgrade
> permanent (no rollback possible after this).
>
>
> On Thu, Nov 22, 2012 at 1:49 PM, yogesh dhari <yogeshdhari@live.com>wrote:
>
>  Thanks Uma,
>
> I used command
> hadoop namenode -upgrade and its started well but got stuck at one point.
>
> 12/11/22 13:06:19 INFO mortbay.log: Started
> SelectChannelConnector@localhost:50070
> 12/11/22 13:06:19 INFO namenode.NameNode: Web-server up at: localhost:50070
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server Responder: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server listener on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 0 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 1 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 2 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 3 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 4 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 5 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 6 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 7 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 8 on 9000: starting
> 12/11/22 13:06:19 INFO ipc.Server: IPC Server handler 9 on 9000: starting
>
> from this point its not showing any progress for past 30 + mins...
>
>
> and Web ui shows
>
> NameNode 'localhost:9000'  Started: Thu Nov 22 13:06:17 IST 2012 Version: 1.0.4,
> r1393290  Compiled: Wed Oct 3 05:13:58 UTC 2012 by hortonfo  Upgrades: Upgrade
> for version -32 has been completed. Upgrade is not finalized.
>
> Please suggest
>
> Regards
> Yogesh Kumar
>
>
>
>
> ------------------------------
> From: maheswara@huawei.com
> To: user@hadoop.apache.org
> Subject: RE: HADOOP UPGRADE ISSUE
> Date: Thu, 22 Nov 2012 07:05:51 +0000
>
>
>  start-all.sh will not carry any arguments to pass to nodes.
> Start with start-dfs.sh
> or start directly namenode with upgrade option. ./hadoop namenode -upgrade
>
> Regards,
> Uma
>  ------------------------------
> *From:* yogesh dhari [yogeshdhari@live.com]
> *Sent:* Thursday, November 22, 2012 12:23 PM
> *To:* hadoop helpforoum
> *Subject:* HADOOP UPGRADE ISSUE
>
>   Hi All,
>
> I am trying upgrading apache *hadoop-0.20.2 to hadoop-1.0.4*.
> I have give same dfs.name.dir, etc as same in hadoop-1.0.4' conf files as
> were in hadoop-0.20.2.
> Now I am starting dfs n mapred using
>
> *start-all.sh -upgrade*
>
> but *namenode *and *datanode* fail to run.
>
> 1) *Namenode's *logs shows::
>
> ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> java.io.IOException:
> File system image contains an old layout version -18.
> An upgrade to version -32 is required.
> Please restart NameNode with -upgrade option.
> .
> .
> ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.io.IOException:
> File system image contains an old layout version -18.
> An upgrade to version -32 is required.
> Please restart NameNode with -upgrade option.
>
>
> 2)* Datanode's* logs shows::
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
> dfs.data.dir: Incorrect permission for /opt/hadoop_newdata_dirr, expected:
> rwxr-xr-x, while actual: rwxrwxrwx
> ****(  how these file permission showing warnings)*****
>
> 2012-11-22 12:05:21,157 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
> dfs.data.dir are invalid.
>
> Please suggest
>
> Thanks & Regards
> Yogesh Kumar
>
>
>
>
>
> --
> Harsh J
>
>
>
>
> --
> Harsh J
>



-- 
Harsh J

Mime
View raw message