hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From daemeon reiydelle <daeme...@gmail.com>
Subject Re: Hadoop Installation on Multihomed Networks
Date Tue, 11 Nov 2014 16:58:19 GMT
You may want to consider configuring host names that embed the subnet in
the host name itself (e.g. foo50, foo40, for foo via each of the 50 and 40
subnets). ssh key file contents & etc may have to be fiddled with a bit









*.......“The race is not to the swift,nor the battle to the strong,but to
those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Nov 11, 2014 at 7:47 AM, Fei Hu <hufei68@gmail.com> wrote:

> Hi,
>
> I am trying to install Hadoop1.0.4 on multihomed networks. I have done it
> as the link
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html.
> But it still doesn’t work.
>
> The datanode could not work. Its log is as the following. In my
> hdfs-site.xml, I set the ip:10.50.0.10 for datanode, but in the following
> report, host = yngcr10nc01/10.10.0.10.I think it may be because in
> /etc/hosts file, I add the pair of ip and hostname:10.10.0.10 YNGCR10NC01
> before.
>
> The problem now is that I could not add 10.50.0.10 YNGCR10NC01 into hosts
> file, because 10.10.0.10 YNGCR10NC01 is necessary for another program.
>
> Is there any way to solve the problem on multihomed networks?
>
> Thanks in advance,
> Fei Hu
>
>
>
>
>
> 2014-11-11 04:20:28,228 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = yngcr10nc01/10.10.0.10
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 1.0.4
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
> ************************************************************/
> 2014-11-11 04:20:28,436 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2014-11-11 04:20:28,446 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2014-11-11 04:20:28,447 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2014-11-11 04:20:28,447 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2014-11-11 04:20:28,572 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2014-11-11 04:20:28,607 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2014-11-11 04:20:29,870 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 0 time(s).
> 2014-11-11 04:20:30,871 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 1 time(s).
> 2014-11-11 04:20:31,872 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 2 time(s).
> 2014-11-11 04:20:32,873 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 3 time(s).
> 2014-11-11 04:20:33,874 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 4 time(s).
> 2014-11-11 04:20:34,875 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 5 time(s).
> 2014-11-11 04:20:35,876 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 6 time(s).
> 2014-11-11 04:20:36,877 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 7 time(s).
> 2014-11-11 04:20:37,877 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 8 time(s).
> 2014-11-11 04:20:38,879 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: yngcr11hm01/10.50.0.5:9000. Already tried 9 time(s).
> 2014-11-11 04:20:38,884 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
> to yngcr11hm01/10.50.0.5:9000 failed on local exception:
> java.net.NoRouteToHostException: No route to host
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
> at org.apache.hadoop.ipc.Client.call(Client.java:1075)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at com.sun.proxy.$Proxy5.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:370)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:429)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:331)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:296)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:356)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
> Caused by: java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:597)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
> at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
> at org.apache.hadoop.ipc.Client.call(Client.java:1050)
> ... 14 more
>
> 2014-11-11 04:20:38,887 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at yngcr10nc01/10.10.0.10
> ************************************************************/
>
> ifconfig:
> brpriv    Link encap:Ethernet  HWaddr EC:F4:BB:C4:86:2A
>           inet addr:10.100.0.10  Bcast:10.100.255.255  Mask:255.255.0.0
>           inet6 addr: fe80::eef4:bbff:fec4:862a/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:1493 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:73898 (72.1 KiB)  TX bytes:1068 (1.0 KiB)
>
> brpub     Link encap:Ethernet  HWaddr EC:F4:BB:C4:86:28
>           inet addr:10.10.0.10  Bcast:10.10.255.255  Mask:255.255.0.0
>           inet6 addr: fe80::eef4:bbff:fec4:8628/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:1274476 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:1013239 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:1488923684 (1.3 GiB)  TX bytes:1142276754 (1.0 GiB)
>
> brpub:1   Link encap:Ethernet  HWaddr EC:F4:BB:C4:86:28
>           inet addr:10.10.3.10  Bcast:10.10.255.255  Mask:255.255.0.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>
> em1       Link encap:Ethernet  HWaddr EC:F4:BB:C4:86:28
>           inet6 addr: fe80::eef4:bbff:fec4:8628/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:6676964 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:1337128 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:5882865885 (5.4 GiB)  TX bytes:1163653184 (1.0 GiB)
>
> em2       Link encap:Ethernet  HWaddr EC:F4:BB:C4:86:2A
>           inet6 addr: fe80::eef4:bbff:fec4:862a/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:16608 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1004640 (981.0 KiB)  TX bytes:1356 (1.3 KiB)
>
> em3       Link encap:Ethernet  HWaddr EC:F4:BB:C4:86:2C
>           inet addr:10.50.0.10  Bcast:10.50.0.255  Mask:255.255.255.0
>           inet6 addr: fe80::eef4:bbff:fec4:862c/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:132716 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:103082 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:78672375 (75.0 MiB)  TX bytes:14882729 (14.1 MiB)
>           Memory:db680000-db700000
>
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:974 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:974 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:118148 (115.3 KiB)  TX bytes:118148 (115.3 KiB)
>
> p5p1      Link encap:Ethernet  HWaddr 00:0A:F7:59:34:B0
>           inet addr:10.30.0.10  Bcast:10.30.255.255  Mask:255.255.0.0
>           inet6 addr: fe80::20a:f7ff:fe59:34b0/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:39190 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:15408562 (14.6 MiB)  TX bytes:956 (956.0 b)
>           Interrupt:40 Memory:d1000000-d17fffff
>
> p5p2      Link encap:Ethernet  HWaddr 00:0A:F7:59:34:B2
>           inet addr:10.30.2.10  Bcast:10.30.255.255  Mask:255.255.0.0
>           inet6 addr: fe80::20a:f7ff:fe59:34b2/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:192233 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:12305366 (11.7 MiB)  TX bytes:606 (606.0 b)
>           Interrupt:44 Memory:d2000000-d27fffff
>
> virbr0    Link encap:Ethernet  HWaddr 52:54:00:12:84:4C
>           inet addr:192.168.122.1  Bcast:192.168.122.255
> Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>
>
>

Mime
View raw message