hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bc Wong (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
Date Fri, 30 Apr 2010 20:50:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862854#action_12862854
] 

bc Wong commented on HDFS-941:
------------------------------

The variance is large on the tests. But they show that the patch isn't slower than trunk.
Tests executed on a 5 node cluster:

* TestDFSIO -read -fileSize 512 -bufferSize 4096 -nrFiles 10

||-||trunk||patched||
|Num trials|6|5|
|Throughput (MB/s)|92|93|
|Avg IO (MB/s)|150|134|
|Std dev|122|77|

* TestDFSIO -read -fileSize 512 -bufferSize 4096 -nrFiles 20

||-||trunk||patched||
|Num trials|5|5|
|Throughput (MB/s)|78|83|
|Avg IO (MB/s)|114|121|
|Std dev|75|76|

* Distributed {{bin/hadoop fs -cat /benchmarks/TestDFSIO/io_data/test_io_$i > /dev/null}},
for i in [0,9]

||-||trunk||patched||
|Num trials|5|5|
|Avg time (sec)|47.8|48.0|
|Std dev|4.2|3.6|

> Datanode xceiver protocol should allow reuse of a connection
> ------------------------------------------------------------
>
>                 Key: HDFS-941
>                 URL: https://issues.apache.org/jira/browse/HDFS-941
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: bc Wong
>         Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch
>
>
> Right now each connection into the datanode xceiver only processes one operation.
> In the case that an operation leaves the stream in a well-defined state (eg a client
reads to the end of a block successfully) the same connection could be reused for a second
operation. This should improve random read performance significantly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message