hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7678) Erasure coding: DFSInputStream with decode functionality
Date Tue, 05 May 2015 04:28:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527883#comment-14527883

Zhe Zhang commented on HDFS-7678:

Thanks Jing for the helpful review. I tested 007 patch without HDFS-8281 and it passes the
tests. There must be some hidden conflicts I missed in the rebase.

     int idx = (int) (((blkStartOffset - lb.getStartOffset()) / cellSize)
-        % dataBlkNum);
+        % (dataBlkNum + parityBlkNum));

Good catch! The root of this bug is that {{StripedBlockUtil#constructInternalBlock}} assigns
the starting offset of a parity block as {{bg.getStartOffset() + idxInBlockGroup * cellSize}}.
 The challenge is that {{DFSInputStream#actualGetFromOneDataNode}} keeps refreshing block
location at each retry loop. I'll make this change.

> Erasure coding: DFSInputStream with decode functionality
> --------------------------------------------------------
>                 Key: HDFS-7678
>                 URL: https://issues.apache.org/jira/browse/HDFS-7678
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7285
>            Reporter: Li Bo
>            Assignee: Zhe Zhang
>         Attachments: BlockGroupReader.patch, HDFS-7678-HDFS-7285.002.patch, HDFS-7678-HDFS-7285.003.patch,
HDFS-7678-HDFS-7285.004.patch, HDFS-7678-HDFS-7285.005.patch, HDFS-7678-HDFS-7285.006.patch,
HDFS-7678-HDFS-7285.007.patch, HDFS-7678-HDFS-7285.008.patch, HDFS-7678.000.patch, HDFS-7678.001.patch
> A block group reader will read data from BlockGroup no matter in striping layout or contiguous
layout. The corrupt blocks can be known before reading(told by namenode), or just be found
during reading. The block group reader needs to do decoding work when some blocks are found

This message was sent by Atlassian JIRA

View raw message