hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8901) Use ByteBuffer in striping positional read
Date Wed, 18 Nov 2015 03:03:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010107#comment-15010107

Kai Zheng commented on HDFS-8901:

bq. There's no bytebuffer version of pread API.
Yeah right. For the byte array version, the passed byte array will be wrapped into a heap
bytebuffer. When Java coder is used, then the heap bytebuffer will match the bytebuffer type
the coder desires.

bq. We can also encode many times stripe by stripe like stateful read, then data copy isn't
In your illustration, given the buf is of 64k 64k 64k 64k 64k 64k 64k 64k 64k ... (+9 cells),
in this change, the whole buffer is divided into +9 cells wrapping into +9 bytebuffers, these
bytebuffers can be referenced into *decodeInputs* and used for decoding. These derived buffers
are not affected in content. The +9 cells are divided into 3 strips, each strip being decoded
a time. Is there anything missed here? Please clarify. Thanks.

bq. We are not pursuing fewer times of data copy since the performance is bounded by IO speed
and coding speed.
Indicated by our latest benchmark tests, coding computation isn't the concern. In high speed
network and disk, in some case (before hitting the limit) IO may also not be the bottleneck.
I agree data copy isn't the key goal we're working here, I just did it and don't find the
overhead in the way.

> Use ByteBuffer in striping positional read
> ------------------------------------------
>                 Key: HDFS-8901
>                 URL: https://issues.apache.org/jira/browse/HDFS-8901
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HDFS-8901-v2.patch, initial-poc.patch
> Native erasure coder prefers to direct ByteBuffer for performance consideration. To prepare
for it, this change uses ByteBuffer through the codes in implementing striping position read.
It will also fix avoiding unnecessary data copying between striping read chunk buffers and
decode input buffers.

This message was sent by Atlassian JIRA

View raw message