hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trevor Robinson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3529) Use direct buffers for data in write path
Date Sun, 28 Oct 2012 23:45:13 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Trevor Robinson updated HDFS-3529:

    Attachment: HDFS-3529.patch

The attached patch is based on [Todd Lipcon's patch|https://github.com/toddlipcon/hadoop-common/tree/trunk-write-pipeline-fast],
but was modified significantly to pass all unit tests and to merge with the datanode encryption
(HDFS-3637) changes.

By switching to direct buffers, the use of native CRC is automatically enabled in {{DataChecksum.verifyChunkedSums}}

On ARM systems, I consistently see about a 10% improvement in TestDFSIO write throughput;
on x86, it varies more, but the average so far is 4%. Read throughput seemed slightly higher,
but was within the run-to-run deviation. Obviously, hardware and tuning configurations vary,
so hopefully others will try it out and share their results. The patch is based on trunk revision
> Use direct buffers for data in write path
> -----------------------------------------
>                 Key: HDFS-3529
>                 URL: https://issues.apache.org/jira/browse/HDFS-3529
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, performance
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: HDFS-3529.patch
> The write path currently makes several unnecessary data copies in order to go to and
from byte arrays. We can improve performance by using direct byte buffers to avoid the copy.
This is also a prerequisite for native checksum calculation (HDFS-3528)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message