hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
Date Tue, 06 Nov 2012 18:34:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491686#comment-13491686

Gopal V commented on HDFS-3529:

I ran through some benchmarks with this patch on (and no tests failed on test-patch).

On a spinning disk it barely made any difference in execution time, but it does speed up DFS
write throughput by a couple of percentage points when I backed HDFS with an SSD with I/O
elevators on deadline.

But for my benchmarks, the bottleneck for the operations are on the client right now - the
client DataStreamer could not write enough data to saturate the SSD on a single thread.

Adding a native chunked checksum generator similar to HDFS-3528 (well, generate is just verify
without a check) on the client code would be worthwhile if this patch needs to actually improve
the big picture of execution - otherwise it might only benefit the replication scenarios slightly.
> Use direct buffers for data in write path
> -----------------------------------------
>                 Key: HDFS-3529
>                 URL: https://issues.apache.org/jira/browse/HDFS-3529
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, performance
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Trevor Robinson
>         Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
> The write path currently makes several unnecessary data copies in order to go to and
from byte arrays. We can improve performance by using direct byte buffers to avoid the copy.
This is also a prerequisite for native checksum calculation (HDFS-3528)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message