hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8319) Erasure Coding: support decoding for stateful read
Date Thu, 04 Jun 2015 00:15:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571877#comment-14571877
] 

Kai Zheng commented on HDFS-8319:
---------------------------------

Thanks Jing for the update.
bq. We use a direct ByteBuffer as buffer for the stripe, while for parity data we use non-direct
ByteBuffer. But the current decoder code only checks the first input to determine whether
using direct bytebuffer API. Thus I updated the decoder code and made it use direct bytebuffer
API if there is at least one direct buffer in inputs/outputs.
In current codec framework it assumes all the inputs and outputs are either on-heap buffers
or direct buffers, no mixing of the both being allowed. This assumption would simplify the
implementation for coders. Basically, Java coders use on-heap buffers that's converted into
bytes array, native coders use direct buffers, for the performance sake. Though for a coder
it will also support other buffer type, but may suffer performance loss. In the coder interface,
{{preferDirectBuffer}} can be used by the caller to see what buffer type would be preferred
as better. Using the same buffer type (no mixing) will also make the buffer allocation and
management easier overall. I'm wondering could we avoid the mixing?
bq. the decoder code uses ByteBuffer#array() for non-direct bytebuffer cases. This can cause
issue if inputs/outputs are derived from slices of the same underlying buffer.
I thought it's easy to support this by some double check and small fixing. Did you see any
problem doing that way? I remembered buffer's position and limit are used, even the underlying
arrays are the same.

> Erasure Coding: support decoding for stateful read
> --------------------------------------------------
>
>                 Key: HDFS-8319
>                 URL: https://issues.apache.org/jira/browse/HDFS-8319
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-8319.001.patch, HDFS-8319.002.patch, HDFS-8319.003.patch
>
>
> HDFS-7678 adds the decoding functionality for pread. This jira plans to add decoding
to stateful read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message