Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 6 May 2011 12:00:05 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1946419032.27975.1304683205310.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-1880) When Namenode network is unplugged,
 DFSClient operations waits for ever
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029904#comment-13029904 ] 

ramkrishna.s.vasudevan commented on HDFS-1880:
----------------------------------------------

Hi pls find the logs below

2010-06-06 19:56:45,406 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
2010-06-06 19:56:45,426 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
2010-06-06 19:56:45,428 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
2010-06-06 19:56:45,433 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(linux112:50010, storageID=, infoPort=50075, ipcPort=50020)
2010-06-06 19:56:45,437 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2010-06-06 19:56:47,804 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: New storage id DS-1238806821-10.18.52.112-50010-1275834407685 is assigned to data-node 10.18.52.112:50010
2010-06-06 19:56:47,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.18.52.112:50010, storageID=DS-1238806821-10.18.52.112-50010-1275834407685, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/home/ramkrishna/opensrchadoop/hadoop-common-0.23.0-SNAPSHOT/hadoop-root/dfs/data/current/finalized'}
2010-06-06 19:56:47,806 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2010-06-06 19:56:47,808 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2010-06-06 19:56:47,808 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: starting
2010-06-06 19:56:47,809 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: starting
2010-06-06 19:56:47,809 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL of 60000msec Initial delay: 0msec
2010-06-06 19:56:47,810 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: starting
2010-06-06 19:56:47,839 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 2 msec to generate and 17 msecs for RPC and NN processing
2010-06-06 19:56:47,840 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block scanner.
2010-06-06 19:57:32,878 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 0 msec to generate and 4 msecs for RPC and NN processing
2010-06-06 19:58:32,953 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 0 msec to generate and 3 msecs for RPC and NN processing
2010-06-06 20:14:40,742 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to /10.18.52.181:9000 failed on local exception: java.io.IOException: No route to host
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
	at org.apache.hadoop.ipc.Client.call(Client.java:1055)
	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
	at $Proxy4.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No route to host
	at sun.nio.ch.FileDispatcher.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
	at sun.nio.ch.IOUtil.read(IOUtil.java:206)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:59)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:132)
	at java.io.FilterInputStream.read(FilterInputStream.java:116)
	at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:371)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:784)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:722)

2010-06-06 20:14:44,748 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:14:48,756 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:14:52,765 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:14:56,773 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:15:00,781 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:15:04,789 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:15:08,798 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:15:12,806 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:15:16,814 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:15:20,822 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:15:23,827 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to /10.18.52.181:9000 failed on local exception: java.net.NoRouteToHostException: No route to host
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
	at org.apache.hadoop.ipc.Client.call(Client.java:1055)
	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
	at $Proxy4.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
	at org.apache.hadoop.ipc.Client.call(Client.java:1032)
	... 5 more

2010-06-06 20:15:27,835 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:15:31,843 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:15:35,851 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:15:39,860 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:15:43,868 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:15:47,876 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:15:51,884 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:15:55,893 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:15:59,901 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:16:03,909 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:16:06,914 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to /10.18.52.181:9000 failed on local exception: java.net.NoRouteToHostException: No route to host
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
	at org.apache.hadoop.ipc.Client.call(Client.java:1055)
	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
	at $Proxy4.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
	at org.apache.hadoop.ipc.Client.call(Client.java:1032)
	... 5 more

2010-06-06 20:16:10,922 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:16:14,930 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:16:18,938 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:16:22,946 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:16:26,955 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:16:30,963 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:16:34,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:16:38,979 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:16:42,988 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:16:46,996 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:16:50,001 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to /10.18.52.181:9000 failed on local exception: java.net.NoRouteToHostException: No route to host
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
	at org.apache.hadoop.ipc.Client.call(Client.java:1055)
	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
	at $Proxy4.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
	at org.apache.hadoop.ipc.Client.call(Client.java:1032)
	... 5 more

2010-06-06 20:16:54,008 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:16:58,017 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:17:02,025 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:17:06,033 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:17:10,041 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:17:14,050 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:17:18,058 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:17:22,066 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:17:26,074 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:17:30,083 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:17:33,088 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to /10.18.52.181:9000 failed on local exception: java.net.NoRouteToHostException: No route to host
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
	at org.apache.hadoop.ipc.Client.call(Client.java:1055)
	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
	at $Proxy4.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
	at org.apache.hadoop.ipc.Client.call(Client.java:1032)
	... 5 more

2010-06-06 20:17:37,095 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:17:41,103 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:17:45,111 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:17:49,120 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:17:53,128 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:17:57,136 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:18:01,144 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:18:04,163 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
2010-06-06 20:18:04,178 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 0 msec to generate and 8 msecs for RPC and NN processing


> When Namenode network is unplugged, DFSClient operations waits for ever
> -----------------------------------------------------------------------
>
>                 Key: HDFS-1880
>                 URL: https://issues.apache.org/jira/browse/HDFS-1880
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>            Reporter: Uma Maheswara Rao G
>
> When NN/DN is shutdown gracefully, the DFSClient operations which are waiting for a response from NN/DN, will throw exception & come out quickly
> But when the NN/DN network is unplugged, the DFSClient operations which are waiting for a response from NN/DN, waits for ever.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira