hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8668) Erasure Coding: revisit buffer used for encoding and decoding.
Date Thu, 11 Aug 2016 13:43:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417261#comment-15417261
] 

Kai Zheng commented on HDFS-8668:
---------------------------------

Thanks for adding some tests. 

Oops, found some issues. Please double check. Thanks!
{code}
-        sum.calculateChunkedSums(buffer.array(), 0, len, checksumBuf, 0);
+        if (buffer.isDirect()) {
+          ByteBuffer directCheckSumBuf =
+              BUFFER_POOL.getBuffer(true, checksumBuf.length);
+          sum.calculateChunkedSums(buffer, directCheckSumBuf);
+          directCheckSumBuf.get(checksumBuf);
+        } else {
+          sum.calculateChunkedSums(buffer.array(), 0, len, checksumBuf, 0);
+        }
+
{code}
{{directCheckSumBuf}} should be put back afterwards. Similarly for the following:
{code}
-    stripedWriter.getChecksum().calculateChunkedSums(
-        targetBuffer.array(), 0, targetBuffer.remaining(),
-        stripedWriter.getChecksumBuf(), 0);
+    if(targetBuffer.isDirect()) {
+      ByteBuffer directCheckSumBuf =
+          ByteBuffer.allocateDirect(stripedWriter.getChecksumBuf().length);
+      stripedWriter.getChecksum().calculateChunkedSums(
+          targetBuffer, directCheckSumBuf);
+      directCheckSumBuf.get(stripedWriter.getChecksumBuf());
+    }else{
+      stripedWriter.getChecksum().calculateChunkedSums(
+          targetBuffer.array(), 0, targetBuffer.remaining(),
+          stripedWriter.getChecksumBuf(), 0);
+    }
{code}
Please also fix the code formats around keywords {{if}}, {{else}}, needing a space.

Unnecessary blank line in the following block:
{code}
   void close() {
+
{code}

> Erasure Coding: revisit buffer used for encoding and decoding.
> --------------------------------------------------------------
>
>                 Key: HDFS-8668
>                 URL: https://issues.apache.org/jira/browse/HDFS-8668
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Yi Liu
>            Assignee: SammiChen
>         Attachments: HDFS-8668-v1.patch, HDFS-8668-v2.patch, HDFS-8668-v3.patch, HDFS-8668-v4.patch,
HDFS-8668-v5.patch, HDFS-8668-v6.patch, HDFS-8668-v7.patch, HDFS-8668-v8.patch
>
>
> For encoding and decoding buffers, currently some places use java heap ByteBuffer,  some
use direct byteBUffer, and some use java byte array.  If the coder implementation is native,
we should use direct ByteBuffer. This jira is to  revisit all encoding/decoding buffers and
improve them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message