hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7678) Erasure coding: DFSInputStream with decode functionality (pread)
Date Mon, 11 May 2015 10:57:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537815#comment-14537815
] 

Kai Zheng commented on HDFS-7678:
---------------------------------

Just checked the place to call the decoder, found the following codes in {{decodeAndFillBuffer}}:
{code}
+    byte[][] outputs = new byte[parityBlkNum][(int) alignedStripe.getLength()];
+    RSRawDecoder rsRawDecoder = new RSRawDecoder();
+    rsRawDecoder.initialize(dataBlkNum, parityBlkNum, (int) alignedStripe.getLength());
+    rsRawDecoder.decode(decodeInputs, decodeIndices, outputs);
{code}
1. Better to declare the variable as {{RawDecoder decoder}};
2. Please create and initialize {{decoder}} during initialization place where a schema is
determined. Doing it per decode call is expensive as in the underlying it may involve preparing
for many coding buffers.
3. By the way, please note with the work in HADOOP-11938 it will be possible to pass the dest
buffers directly to the decode call as output buffers so we will avoid a data copy thereafter.

> Erasure coding: DFSInputStream with decode functionality (pread)
> ----------------------------------------------------------------
>
>                 Key: HDFS-7678
>                 URL: https://issues.apache.org/jira/browse/HDFS-7678
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7285
>            Reporter: Li Bo
>            Assignee: Zhe Zhang
>         Attachments: BlockGroupReader.patch, HDFS-7678-HDFS-7285.002.patch, HDFS-7678-HDFS-7285.003.patch,
HDFS-7678-HDFS-7285.004.patch, HDFS-7678-HDFS-7285.005.patch, HDFS-7678-HDFS-7285.006.patch,
HDFS-7678-HDFS-7285.007.patch, HDFS-7678-HDFS-7285.008.patch, HDFS-7678-HDFS-7285.009.patch,
HDFS-7678-HDFS-7285.010.patch, HDFS-7678-HDFS-7285.012.patch, HDFS-7678-HDFS-7285.013.patch,
HDFS-7678.000.patch, HDFS-7678.001.patch
>
>
> A block group reader will read data from BlockGroup no matter in striping layout or contiguous
layout. The corrupt blocks can be known before reading(told by namenode), or just be found
during reading. The block group reader needs to do decoding work when some blocks are found
corrupt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message