hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7678) Erasure coding: DFSInputStream with decode functionality
Date Tue, 05 May 2015 01:18:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527716#comment-14527716
] 

Jing Zhao commented on HDFS-7678:
---------------------------------

Thanks for the great work, Zhe! I have not finished my review yet, but looks like the latest
008 patch will fail TestDFSStripedInputStream (and it passed the pread part but failed in
the stateful read part)? Besides, for {{ReadPortion}}'s public fields, I changed them to private
because they both are not declared as final and can be modified outside. Thus feels like to
restrict the modification going through the setters will make it easier to track. But not
feeling very strong on this.

Another quick comment is, the following change looks incorrect to me. For example, if {{blkStartOffset
- lb.getStartOffset()}} points to cellSize * 10 inside of the block group, we should read
from the 5th data block instead of 2nd.
{code}
     int idx = (int) (((blkStartOffset - lb.getStartOffset()) / cellSize)
-        % dataBlkNum);
+        % (dataBlkNum + parityBlkNum));
{code}

Will try to finish the review and post comments later today.

> Erasure coding: DFSInputStream with decode functionality
> --------------------------------------------------------
>
>                 Key: HDFS-7678
>                 URL: https://issues.apache.org/jira/browse/HDFS-7678
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7285
>            Reporter: Li Bo
>            Assignee: Zhe Zhang
>         Attachments: BlockGroupReader.patch, HDFS-7678-HDFS-7285.002.patch, HDFS-7678-HDFS-7285.003.patch,
HDFS-7678-HDFS-7285.004.patch, HDFS-7678-HDFS-7285.005.patch, HDFS-7678-HDFS-7285.006.patch,
HDFS-7678-HDFS-7285.007.patch, HDFS-7678-HDFS-7285.008.patch, HDFS-7678.000.patch, HDFS-7678.001.patch
>
>
> A block group reader will read data from BlockGroup no matter in striping layout or contiguous
layout. The corrupt blocks can be known before reading(told by namenode), or just be found
during reading. The block group reader needs to do decoding work when some blocks are found
corrupt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message