hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
Date Thu, 28 Aug 2014 08:36:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113546#comment-14113546
] 

Yi Liu commented on HDFS-3529:
------------------------------

Hi [~scurrilous], could I take this JIRA and continue the work?

Currently HDFS cache is supported, and memory support is ongoing, so I think this will be
a good improvement. 

Of course if use HDD, the benefit we get from saving bytes copy by using direct buffer is
very small compared to the Disk IO.

> Use direct buffers for data in write path
> -----------------------------------------
>
>                 Key: HDFS-3529
>                 URL: https://issues.apache.org/jira/browse/HDFS-3529
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, performance
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Trevor Robinson
>         Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
>
>
> The write path currently makes several unnecessary data copies in order to go to and
from byte arrays. We can improve performance by using direct byte buffers to avoid the copy.
This is also a prerequisite for native checksum calculation (HDFS-3528)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message