hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6569) OOB massage can't be sent to the client when DataNode shuts down for upgrade
Date Thu, 19 Jun 2014 18:34:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037640#comment-14037640
] 

Brandon Li commented on HDFS-6569:
----------------------------------

The client stack trace:
{noformat}
14/06/19 11:26:23 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block
BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1997)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:793)
14/06/19 11:26:23 WARN hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcher.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
	at sun.nio.ch.IOUtil.write(IOUtil.java:40)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:271)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:574)
put: All datanodes 127.0.0.1:50010 are bad. Aborting...
{noformat}

The DataNode error:
{noformat}
2014-06-19 11:26:23,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Shutting down
for restart (BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004).
2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Sending an out
of band ack of type OOB_RESTART
2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Error sending
OOB Ack.
java.io.IOException: The stream is closed
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1339)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendOOBResponse(BlockReceiver.java:1041)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:802)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
        at java.lang.Thread.run(Thread.java:695)

{noformat}


> OOB massage can't be sent to the client when DataNode shuts down for upgrade
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-6569
>                 URL: https://issues.apache.org/jira/browse/HDFS-6569
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 3.0.0, 2.2.0
>            Reporter: Brandon Li
>
> The socket is closed too early before the OOB message can be sent to client, which causes
the write pipeline failure.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message