hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
Date Tue, 21 Jun 2016 16:58:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342175#comment-15342175
] 

Kai Zheng commented on HDFS-10460:
----------------------------------

bq. can continue setting block.setNumBytes(getRemaining()); logic in Replicated and Striped
block. Then will consider reconstruction as a special case and will create reconBlockGroup
object with actualNumBytes, like I'm doing in the current patch. 
Yeah I understand your opinion. I thought both approaches should work because they're essentially
the same. You're looking at this from another perspective and want this to be consistent with
replicated block. It sounds good. On the other hand, checksum for striping block group is
a newly added function and protocol, and in this, we need to pass the exact block group object
to datanode side for reconstruction need, as we did for erasure coding worker, and to do the
checksuming in datanode side, we also need to pass the requestedLength. Doing this would look
the related codes and protocol look more naturable and readable. IMO, {{requestedLength}}
would be better understandable than {{actualNumBytes}}, and for the latter, you would have
to explain about it, and you also need to setNumBytes back and forth against the block group
object.

> Erasure Coding: Recompute block checksum for a particular range less than file size on
the fly by reconstructing missed block
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10460
>                 URL: https://issues.apache.org/jira/browse/HDFS-10460
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-10460-00.patch, HDFS-10460-01.patch
>
>
> This jira is HDFS-9833 follow-on task to address reconstructing block and then recalculating
block checksum for a particular range query.
> For example,
> {code}
> // create a file 'stripedFile1' with fileSize = cellSize * numDataBlocks = 65536 * 6
= 393216
> FileChecksum stripedFileChecksum = getFileChecksum(stripedFile1, 10, true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message