hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bc Wong (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
Date Fri, 19 Mar 2010 06:41:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847244#action_12847244

bc Wong commented on HDFS-941:

Thanks for the review, Todd. I'll add more tests, and look into making a cache of size >

bq. I think there is a concurrency issue here. Namely, the positional read API calls through
into fetchBlockByteRange, which will use the existing cached socket, regardless of other concurrent
operations. So we may end up with multiple block readers on the same socket and everything
will fall apart.

That should be fine. Each {{SocketCacheEntry}} has a unique {{Socket}}, owned by its {{BlockReader}}.
One of the reuse condition is that the {{BlockReader}} has finished reading on that {{Socket}}
({{hasConsumedAll()}}). Note that we do not reuse {{BlockReader}}. So at this point, it should
be safe to take the {{Socket}} away from its previous owner and give it to a new {{BlockReader}}.

I'll add tests for this though.

> Datanode xceiver protocol should allow reuse of a connection
> ------------------------------------------------------------
>                 Key: HDFS-941
>                 URL: https://issues.apache.org/jira/browse/HDFS-941
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: bc Wong
>         Attachments: HDFS-941-1.patch
> Right now each connection into the datanode xceiver only processes one operation.
> In the case that an operation leaves the stream in a well-defined state (eg a client
reads to the end of a block successfully) the same connection could be reused for a second
operation. This should improve random read performance significantly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message