hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trevor Robinson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
Date Wed, 07 Nov 2012 01:16:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492024#comment-13492024

Trevor Robinson commented on HDFS-3529:

My patch should enable native CRC on the datanode side, so it sounds like we just need to
enable it on the client side too. This patch seemed large enough already that I wanted to
get it reviewed and committed before embarking on the next step.

FYI, I swapped 4 HDDs for 4 SSDs in the E3-1240 system I mentioned earlier. On trunk, I get
a TestDFSIO write throughput of 257 MiB/s (96 1.5 GiB files, 4 mappers, 1 node); with the
patch, I get 326 MiB/s (a speedup of 27%). I don't think people generally run Hadoop on SSD,
but just wanted to point out that this patch alone can be significant in some configurations.
> Use direct buffers for data in write path
> -----------------------------------------
>                 Key: HDFS-3529
>                 URL: https://issues.apache.org/jira/browse/HDFS-3529
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, performance
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Trevor Robinson
>         Attachments: dfsio-x86-trunk-vs-3529.png, HDFS-3529.patch
> The write path currently makes several unnecessary data copies in order to go to and
from byte arrays. We can improve performance by using direct byte buffers to avoid the copy.
This is also a prerequisite for native checksum calculation (HDFS-3528)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message