hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trevor Robinson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3529) Use direct buffers for data in write path
Date Tue, 06 Nov 2012 18:08:13 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Trevor Robinson updated HDFS-3529:

    Attachment: dfsio-x86-trunk-vs-3529.png

Attached image shows Ganglia charts from two runs of TestDFSIO (each with Write followed by
Read): first is with this patch, second is without it. The I/O rates are essentially the same
(disk-bound), but the write phase shows 5% lower CPU utilization with the patch. CPU used
is a Sandy Bridge E3-1240 (4 cores, 3.3 GHz).
> Use direct buffers for data in write path
> -----------------------------------------
>                 Key: HDFS-3529
>                 URL: https://issues.apache.org/jira/browse/HDFS-3529
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, performance
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Trevor Robinson
>         Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
> The write path currently makes several unnecessary data copies in order to go to and
from byte arrays. We can improve performance by using direct byte buffers to avoid the copy.
This is also a prerequisite for native checksum calculation (HDFS-3528)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message