hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8901) Use ByteBuffer in striping positional read
Date Tue, 17 Nov 2015 10:54:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008477#comment-15008477
] 

Walter Su commented on HDFS-8901:
---------------------------------

bq. (from description) this change uses ByteBuffer through the codes in implementing striping
position read. It will also fix avoiding unnecessary data copying between striping read chunk
buffers and decode input buffers.
I think, for PositionStripeReader, the data copy from buf\[\](given by caller) to decodeInputs\[\]\[\]
is unavoidable. An {{AlignedStripe}} actually merged multiple real stripes. I mean if cellsize
is 64kb, a real stripe is 6*64kb, an {{AlignedStripe}} could be n*6*64kb (assume given buf\[\]
is larger than that). decodeInputs\[0\]\[n*64k\] is copied from n positions from buf\[\].

I understand native erasure coder prefers direct ByteBuffer. So we can make decodeInputs\[\]
a direct ByteBuffer array. Again we copy data from buf\[\]  to decodeInputs\[\] (off-heap
bytebuffers).

So I think the changes to PositionStripeReader are good, and changes to DFSInputStream are
unnecessary.

> Use ByteBuffer in striping positional read
> ------------------------------------------
>
>                 Key: HDFS-8901
>                 URL: https://issues.apache.org/jira/browse/HDFS-8901
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HDFS-8901-v2.patch, initial-poc.patch
>
>
> Native erasure coder prefers to direct ByteBuffer for performance consideration. To prepare
for it, this change uses ByteBuffer through the codes in implementing striping position read.
It will also fix avoiding unnecessary data copying between striping read chunk buffers and
decode input buffers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message