hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9719) Refactoring ErasureCodingWorker into smaller reusable constructs
Date Thu, 07 Apr 2016 06:16:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229763#comment-15229763
] 

Hudson commented on HDFS-9719:
------------------------------

FAILURE: Integrated in Hadoop-trunk-Commit #9575 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9575/])
HDFS-9719. Refactoring ErasureCodingWorker into smaller reusable (uma.gangumalla: rev 3c18a53cbd2efabb2ad108d63a0b0b558424115f)
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockWriter.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/package-info.java
* hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedWriter.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReader.java


> Refactoring ErasureCodingWorker into smaller reusable constructs
> ----------------------------------------------------------------
>
>                 Key: HDFS-9719
>                 URL: https://issues.apache.org/jira/browse/HDFS-9719
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>             Fix For: 3.0.0
>
>         Attachments: HDFS-9719-v1.patch, HDFS-9719-v2.patch, HDFS-9719-v3.patch, HDFS-9719-v4.patch,
HDFS-9719-v5.patch, HDFS-9719-v6.patch, HDFS-9719-v7.patch, HDFS-9719-v8.patch, HDFS-9719-v9.patch
>
>
> This would suggest and refactor {{ErasureCodingWorker}} into smaller constructs to be
reused in other places like block group checksum computing in datanode side. As discussed
in HDFS-8430 and implemented in HDFS-9694 patch, checksum computing for striped block groups
would be distributed to datanode in the group, where data block data should be able to be
reconstructed when missed/corrupted to recompute the block checksum. The most needed codes
are in the current ErasureCodingWorker and could be reused in order to avoid duplication.
Fortunately, we have very good and complete tests, which would make the refactoring much easier.
The refactoring will also help a lot for subsequent tasks in phase II for non-striping erasure
coded files and blocks. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message