hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10460) Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block
Date Wed, 22 Jun 2016 13:50:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15344331#comment-15344331
] 

Kai Zheng commented on HDFS-10460:
----------------------------------

Thanks for the update, Rakesh. It looks much close now:

1. In StripedBlockChecksumReconstructor#reconstruct(), could we move the most of the new codes
to a new function like {{checksumWithTargetOutput(targetBuffer)}}? Doing this would separate
the checksum logic from the reconstruction stuffs and when we add to handle more failures
than one, the new function can be reused.

2. Ref. the following test case: 1) The 1 requestedLen case is good to have, might also have
a larger one like 10? 2) Don't quite get why we have an array for the only one filepath. 3)
You have quite a few tests like this, guess you could move the codes of preparing for the
test file to a common place.
{code}
+  /**
+   * Test to verify that the checksum can be computed for a small file less than
+   * bytesPerCRC size.
+   */
+  @Test(timeout = 90000)
+  public void testStripedFileChecksumWithMissedDataBlocksRangeQuery16()
+      throws Exception {
+    int fileLength = 100;
+    byte[] fileData = StripedFileTestUtil.generateBytes(fileLength);
+    String stripedFile3 = ecDir + "/stripedFileChecksum3";
+    String[] filePaths = new String[] {stripedFile3};
+
+    for (String filePath : filePaths) {
+      Path testPath = new Path(filePath);
+      DFSTestUtil.writeFile(fs, testPath, fileData);
+    }
+    testStripedFileChecksumWithMissedDataBlocksRangeQuery(stripedFile3, 1);
+  }
{code}


> Erasure Coding: Recompute block checksum for a particular range less than file size on
the fly by reconstructing missed block
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10460
>                 URL: https://issues.apache.org/jira/browse/HDFS-10460
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-10460-00.patch, HDFS-10460-01.patch, HDFS-10460-02.patch, HDFS-10460-03.patch
>
>
> This jira is HDFS-9833 follow-on task to address reconstructing block and then recalculating
block checksum for a particular range query.
> For example,
> {code}
> // create a file 'stripedFile1' with fileSize = cellSize * numDataBlocks = 65536 * 6
= 393216
> FileChecksum stripedFileChecksum = getFileChecksum(stripedFile1, 10, true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message