hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11938) Fix ByteBuffer version encode/decode API of raw erasure coder
Date Mon, 11 May 2015 09:41:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537768#comment-14537768

Kai Zheng commented on HADOOP-11938:

Hi [~hitliuyi],

Thanks for your review and the great comments. They're really helpful to make the solid base.
My following patch will address most of them. Thanks!
bq. What's the reason we need to reset inputbuffer? Input buffers are given by caller.
Yes for now we don't need it. I got it from HADOOP-11847, there it's needed to reset input
buffers provided from reused internal buffers. I will remove it and only add it when really
necessary then.
bq.  all the input have the same length besides some inputs are null
For now all the inputs shall not be null. In HADOOP-11847 some inputs can be null to indicate
not to read those units. I would really add such checks in HADOOP-11847 because it's not trivial
to do such checks. This patch would be better to focus on the ByteBuffer supporting.
bq. We should also enhance the description in RawErasureDecoder#decode
Yes it should . I did it in HADOOP-11847 already. Could we leave it now in this issue?

Thanks a lot!

> Fix ByteBuffer version encode/decode API of raw erasure coder
> -------------------------------------------------------------
>                 Key: HADOOP-11938
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11938
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HADOOP-11938-HDFS-7285-v1.patch, HADOOP-11938-HDFS-7285-workaround.patch
> While investigating a test failure in {{TestRecoverStripedFile}}, one issue in raw erasrue
coder, caused by an optimization in below codes. It assumes the  heap buffer backed by the
bytes array available for reading or writing always starts with zero and takes the whole space.
> {code}
>   protected static byte[][] toArrays(ByteBuffer[] buffers) {
>     byte[][] bytesArr = new byte[buffers.length][];
>     ByteBuffer buffer;
>     for (int i = 0; i < buffers.length; i++) {
>       buffer = buffers[i];
>       if (buffer == null) {
>         bytesArr[i] = null;
>         continue;
>       }
>       if (buffer.hasArray()) {
>         bytesArr[i] = buffer.array();
>       } else {
>         throw new IllegalArgumentException("Invalid ByteBuffer passed, " +
>             "expecting heap buffer");
>       }
>     }
>     return bytesArr;
>   }
> {code} 

This message was sent by Atlassian JIRA

View raw message