hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
Date Wed, 08 Jun 2011 19:45:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046162#comment-13046162
] 

stack commented on HDFS-941:
----------------------------

On occasion I see these new additions to the datanode log:

{code}
2011-06-08 12:37:20,478 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Client did
not send a valid status code after reading. Will close connection.
2011-06-08 12:37:20,480 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Client did
not send a valid status code after reading. Will close connection.
2011-06-08 12:37:20,482 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Client did
not send a valid status code after reading. Will close connection.
2011-06-08 12:37:20,483 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Client did
not send a valid status code after reading. Will close connection.
{code}

Should these be logged as DEBUG and not ERROR?

I see this too, don't think it related:

{code}
2011-06-08 12:40:09,642 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-2049668997072761677_6556 src: /10.4.9.34:36343 dest: /10.4.9.34:10010
2011-06-08 12:40:09,661 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks()
exception: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:204)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:392)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:481)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opReadBlock(DataXceiver.java:237)
        at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opReadBlock(DataTransferProtocol.java:356)
        at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:328)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:169)
        at java.lang.Thread.run(Thread.java:662)
{code}

Odd is that this is machine talking to itself.

> Datanode xceiver protocol should allow reuse of a connection
> ------------------------------------------------------------
>
>                 Key: HDFS-941
>                 URL: https://issues.apache.org/jira/browse/HDFS-941
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: bc Wong
>         Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch,
HDFS-941-4.patch, HDFS-941-5.patch, HDFS-941-6.22.patch, HDFS-941-6.patch, HDFS-941-6.patch,
HDFS-941-6.patch, hdfs-941.txt, hdfs-941.txt, hdfs941-1.png
>
>
> Right now each connection into the datanode xceiver only processes one operation.
> In the case that an operation leaves the stream in a well-defined state (eg a client
reads to the end of a block successfully) the same connection could be reused for a second
operation. This should improve random read performance significantly.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message