hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
Date Fri, 19 Mar 2010 06:47:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847246#action_12847246

Todd Lipcon commented on HDFS-941:

bq. So at this point, it should be safe to take the Socket away from its previous owner and
give it to a new BlockReader.

I don't think this is true. Two issues:

- At DFSInputStream.java:699, we check the cached socket for usability. If we have two threads
concurrently at this line, they can both detect that it's usable, and then both move on to
construct block readers on the same socket. One of them will win the entry.setOwner race at
line 723, but they'll both still be operating on the same socket and generate bad commands.
- If a block reader exists and is not complete, usable() returns false so the socket won't
be reused. But on line 701, it calls entry.close() which calls through to close the underlying
socket. This will destroy the read currently going on.

> Datanode xceiver protocol should allow reuse of a connection
> ------------------------------------------------------------
>                 Key: HDFS-941
>                 URL: https://issues.apache.org/jira/browse/HDFS-941
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: bc Wong
>         Attachments: HDFS-941-1.patch
> Right now each connection into the datanode xceiver only processes one operation.
> In the case that an operation leaves the stream in a well-defined state (eg a client
reads to the end of a block successfully) the same connection could be reused for a second
operation. This should improve random read performance significantly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message