hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doss_IPH <d...@intellipowerhive.com>
Subject Re: Datanode Throwing NoRouteToHostException
Date Wed, 21 Oct 2009 04:46:50 GMT

Comment all the IPs in /etc/hosts first except localhost or 127.0.0.1 as i
have given below. Then, give in conf/hadoop-site.xml
fs.default.name parameter value is hdfs://hdfs://[Master Node IP]:9000/
mapred.job.tracker parameter value is hdfs://[Job Tracker IP]:9001
Make sure you have enabled ssh and sshd running without password.


lazikid wrote:
> 
> I need some help with setting up a Hadoop cluster. The datanode on the
> slave is not coming up throwing java.net.NoRouteToHostException: No route
> to host. Please see the details below.
> 
> I have a centos master and a fedora slave. Both have Java 6 and Hadoop
> 0.20.1. I instaleed hadoop under /opt in both machines.
> 
> The machines can ping one another using hostname and ip. They can also
> make password-less ssh connection to one another using both hostname and
> ip.
> 
> This is what I have in the /etc/hosts file for the master :
> 
> ************************************************
> 127.0.0.1       localhost.localdomain localhost
> #192.168.1.125   centos
> #192.168.1.102   fedora
> ::1             localhost6.localdomain6 localhost6
> ************************************************
> 
> This is what I have in the /etc/hosts for the slave
> 
> ************************************************
> 127.0.0.1	localhost.localdomain localhost
> #192.168.1.104	ubuntu64
> #192.168.1.102	fedora
> #192.168.1.125	centos
> ::1         localhost localhost.localdomain localhost6
> localhost6.localdomain6
> ************************************************
> 
> 
> 
> These are my configuration files :
> 
> 1. core-site.xml 
> 
> <configuration>
> <property>
>   <name>fs.default.name</name>
>   <value>hdfs://centos:54310</value>
>   <description>determine the host, port, etc. for a
> filesystem.</description>
> </property>
> </configuration>
> 
> 2. mapred-site.xml
> 
> <configuration>
> <property>
>   <name>mapred.job.tracker</name>
>   <value>hdfs://centos:54311</value>
>   <description>....</description>
> </property>
> </configuration>
> 
> 3. masters
>   
>    centos
> 
> 4. slaves
> 
>    fedora
> 
> 
> I issued bin/start-dfs.sh from the master(centos) machine and this is what
> I see :
> 
> starting namenode, logging to xxxxxxxxxxx
> fedora: starting datanode, logging to xxxxxxx
> centos: starting secondarynamenode, logging to xxxxxx
> 
> When I checked the slave(fedora) logs, this what I see :
> 
> *************************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = fedora/192.168.1.102
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.1
> STARTUP_MSG:   build =
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> 810220; compiled by 'oom' on Tue Sep  1 20:55:56 UTC 
> 
> 2009
> ************************************************************/
> 2009-10-17 17:05:43,385 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 0 time(s).
> 2009-10-17 17:05:44,387 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 1 time(s).
> 2009-10-17 17:05:45,389 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 2 time(s).
> 2009-10-17 17:05:46,390 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 3 time(s).
> 2009-10-17 17:05:47,388 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 4 time(s).
> 2009-10-17 17:05:48,390 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 5 time(s).
> 2009-10-17 17:05:49,393 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 6 time(s).
> 2009-10-17 17:05:50,394 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 7 time(s).
> 2009-10-17 17:05:51,395 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 8 time(s).
> 2009-10-17 17:05:52,398 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: centos/192.168.1.125:54310. Already tried 9 time(s).
> 2009-10-17 17:05:52,404 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
> to centos/192.168.1.125:54310 failed on local 
> 
> exception: java.net.NoRouteToHostException: No route to host
> 	at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:742)
> 	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> 	at $Proxy4.getProtocolVersion(Unknown Source)
> 	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> 	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
> 	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
> 	at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
> 	at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
> 	at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
> 	at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
> 	at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
> 	at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> 	at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
> 	at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
> Caused by: java.net.NoRouteToHostException: No route to host
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> 	at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> 	at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
> 	at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:719)
> 	... 13 more
> 
> 2009-10-17 17:05:52,407 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at fedora/192.168.1.102
> ************************************************************/
> 
> *************************************************************************
> 
> 
> 
> 
> 
> The log on the master shows :
> 
> 2009-10-17 17:05:39,352 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = centos/192.168.1.125
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.1
> STARTUP_MSG:   build =
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> 810220; compiled by 'oom' on Tue Sep  1 20:55:56 UTC 
> 
> 2009
> ************************************************************/
> 2009-10-17 17:05:39,615 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=NameNode, port=54310
> 2009-10-17 17:05:39,620 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
> centos/192.168.1.125:54310
> 2009-10-17 17:05:39,622 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=NameNode, sessionId=null
> 2009-10-17 17:05:39,624 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
> Initializing NameNodeMeterics using context 
> 
> object:org.apache.hadoop.metrics.spi.NullContext
> 2009-10-17 17:05:39,779 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=joe,joe
> 2009-10-17 17:05:39,780 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
> 2009-10-17 17:05:39,781 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2009-10-17 17:05:39,877 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context 
> 
> object:org.apache.hadoop.metrics.spi.NullContext
> 2009-10-17 17:05:39,883 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2009-10-17 17:05:39,929 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Number of files = 1
> 2009-10-17 17:05:39,933 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Number of files under construction = 0
> 2009-10-17 17:05:39,933 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 97 loaded in 0 seconds.
> 2009-10-17 17:05:39,934 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Edits file /tmp/hadoop-joe/dfs/name/current/edits of size 4 edits # 0 
> 
> loaded in 0 seconds.
> 2009-10-17 17:05:39,972 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 97 saved in 0 seconds.
> 2009-10-17 17:05:40,130 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 469 msecs
> 2009-10-17 17:05:40,131 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of
> blocks = 0
> 2009-10-17 17:05:40,131 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid
> blocks = 0
> 2009-10-17 17:05:40,131 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> under-replicated blocks = 0
> 2009-10-17 17:05:40,132 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of 
> over-replicated blocks = 0
> 2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Leaving safe mode after 0 secs.
> 2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Network topology has 0 racks and 0 datanodes
> 2009-10-17 17:05:40,132 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> UnderReplicatedBlocks has 0 blocks
> 2009-10-17 17:05:40,320 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2009-10-17 17:05:40,518 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. 
> 
> Opening the listener on 50070
> 2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() 
> 
> returned 50070
> 2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
> 2009-10-17 17:05:40,520 INFO org.mortbay.log: jetty-6.1.14
> 2009-10-17 17:05:46,612 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50070
> 2009-10-17 17:05:46,612 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> 0.0.0.0:50070
> 2009-10-17 17:05:46,613 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2009-10-17 17:05:46,615 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
> 2009-10-17 17:05:46,629 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 54310: starting
> 2009-10-17 17:05:46,634 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 54310: starting
> 2009-10-17 17:05:46,636 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 54310: starting
> 2009-10-17 17:05:46,637 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 54310: starting
> 2009-10-17 17:05:46,641 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 54310: starting
> 2009-10-17 17:05:46,642 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 54310: starting
> 2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 54310: starting
> 2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 54310: starting
> 2009-10-17 17:05:46,900 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 54310: starting
> 2009-10-17 17:05:46,916 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 54310: starting
> 2009-10-17 17:11:10,967 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 192.168.1.125
> 2009-10-17 17:11:10,968 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> transactions: 0 Total time for transactions(ms): 0Number of 
> 
> transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 
> 2009-10-17 17:11:11,518 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll FSImage from
> 192.168.1.125
> 2009-10-17 17:11:11,519 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> transactions: 0 Total time for transactions(ms): 0Number of 
> 
> transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 2 
> 
> 
>    
> 

-- 
View this message in context: http://www.nabble.com/Datanode-Throwing-NoRouteToHostException-tp25946798p25986468.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Mime
View raw message