hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2891) Some times first DataNode detected as bad when we power off for the second DataNode.
Date Sat, 04 Feb 2012 09:59:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200372#comment-13200372
] 

Uma Maheswara Rao G commented on HDFS-2891:
-------------------------------------------


More info:

This happens when we test with Hbase.
Run the HDFS and HBase cluster normally, and Suddenly Power-off the mid Datanode from the
pipeline.
Then Clinet was getting following exception.

{noformat}
[2012-01-31 11:15:42,596] [WARN ] [ResponseProcessor for block blk_1327946241860_1109] [org.apache.hadoop.hdfs.DFSClient
3287] DFSOutputStream ResponseProcessor exception for block blk_1327946241860_1109java.net.SocketTimeoutException:
69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/XXX:59179 remote=/FIRST_DATANODE_IP:10010]
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:167)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:131)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3240)
{noformat}

After that client closed the connection and first Datanode was getting EOFException and interrupted
the threads.

{noformat}
[2012-01-31 11:15:42,597] [INFO ] [org.apache.hadoop.hdfs.server.datanode.DataXceiver@3e909a58]
[org.apache.hadoop.hdfs.server.datanode.DataNode 816] Exception in receiveBlock for block
blk_1327946241860_1109
java.io.EOFException: while trying to read 31744 bytes
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:352)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:399)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:611)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:514)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:138)
	at java.lang.Thread.run(Thread.java:662)
[2012-01-31 11:15:42,598] [INFO ] [PacketResponder 2 for Block blk_1327946241860_1109] [org.apache.hadoop.hdfs.server.datanode.DataNode
1123] PacketResponder blk_1327946241860_1109 2 : Thread is interrupted.
[2012-01-31 11:15:42,598] [INFO ] [PacketResponder 2 for Block blk_1327946241860_1109] [org.apache.hadoop.hdfs.server.datanode.DataNode
1194] PacketResponder 2 for block blk_1327946241860_1109 terminating
{noformat}

Finally Client is marking this first DataNode as bad.

{noformat}
[2012-01-31 11:15:42,597] [WARN ] [DataStreamer for file /hbase/.logs/DDB01,20020,1327946260020/DDB01%3A20020.1327978683479
block blk_1327946241860_1109] [org.apache.hadoop.hdfs.DFSClient 3326] Error Recovery for block
blk_1327946241860_1109 bad datanode[0] FIRST_DATANODE_IP:10010
[2012-01-31 11:15:42,597] [WARN ] [DataStreamer for file /hbase/.logs/DDB01,20020,1327946260020/DDB01%3A20020.1327978683479
block blk_1327946241860_1109] [org.apache.hadoop.hdfs.DFSClient 3380] Error Recovery for block
blk_1327946241860_1109 in pipeline FIRST_DATANODE_IP:10010, SECOND_DATANODE_IP:10010, THIRD_DATANODE_IP:10010:
bad datanode FIRST_DATANODE_IP:10010
[2012-01-31 11:15:46,607] [INFO ] [DataStreamer for file /hbase/.logs/DDB01,20020,1327946260020/DDB01%3A20020.1327978683479
block blk_1327946241860_1109] [org.apache.hadoop.ipc.Client 514] Retrying connect to server:
/SECOND_DATANODE_IP:10020. Already tried 0 time(s).
{noformat}

Infact first datanode is healthy, but unportunately due to this timeouts clinet detected first
datanode as bad. Then immediately it is trying with the second datanode( by choosing it as
primary node).

Ideally first Datanode should get first timeout because he will wait only 66000ms for ack
response from second datanode(power off DN). Here clients waits for 69000ms. But for some
reason Client is getting first time out exception that firstDatanode. So, first DN marked
as bad instaed of second one.

Impact is, Clinet again tries for the second datanode and obiously will fail again.It will
6 times unnecessarily it will retry. 

{noformat}
2012-01-31 11:19:58,565] [WARN ] [DataStreamer for file /hbase/.logs/DDB01,20020,1327946260020/DDB01%3A20020.1327978683479
block blk_1327946241860_1109] [org.apache.hadoop.hdfs.DFSClient 3426] Error Recovery for block
blk_1327946241860_1109 failed  because recovery from primary datanode SECOND_DATANODE_IP:10010
failed 6 times.  Pipeline was FIRST_DATANODE_IP:10010, SECOND_DATANODE_IP:10010, THIRD_DATANODE_IP:10010.
Marking primary datanode as bad.
{noformat}

Finally it will end up in writing only one replica in third DataNode. But here we have two
nodes healthy. 
                
> Some times first DataNode detected as bad when we power off for the second DataNode.
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-2891
>                 URL: https://issues.apache.org/jira/browse/HDFS-2891
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node, hdfs client
>    Affects Versions: 1.1.0
>            Reporter: Uma Maheswara Rao G
>
> In one of my clusters, observed this situation.
> This issue looks to be due to time out in ResponseProcesser at client side, it is marking
first DataNode as bad.
> This happens in 20.2 version. This can be there in branch-1 as well and will check for
trunk.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message