hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7369) Erasure coding: distribute recovery work for striped blocks to DataNode
Date Fri, 13 Mar 2015 21:28:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361131#comment-14361131

Kai Zheng commented on HDFS-7369:

Thanks Zhe for the update. Some comments.

1. Looking at some changes like below, I'm wondering if we could have any abstract concept
for the two cases, replication and erasure coding or recovery. With that, something like {{neededReplications}}
can be better named. However I'm not sure how much change will be incurred. You might be thinking
"recovery" can cover "replication", but "recovery" is mainly discussed or mentioned and relevant
in ec context and this issue title.
-   * Scan blocks in {@link #neededReplications} and assign replication
-   * work to data-nodes they belong to.
+   * Scan blocks in {@link #neededReplications} and assign recovery
+   * (replication or erasure coding) work to data-nodes they belong to.
-  int computeReplicationWork(int blocksToProcess) {
+  int computeBlockRecoveryWork(int blocksToProcess) {
2. Why we have this ? Are you using Java 8 ?
-            containingNodes = new ArrayList<DatanodeDescriptor>();
-            List<DatanodeStorageInfo> liveReplicaNodes = new ArrayList<DatanodeStorageInfo>();
+            containingNodes = new ArrayList<>();
+            List<DatanodeStorageInfo> liveReplicaNodes = new ArrayList<>();
3. In the codes for above, could we have initial capacity value for the two array lists ?
4. A minor, clean up.
+//              ErasureCodingWork ecw =
5. Assume we have conclusion about ec policy or unified storage policy in HDFS-7285, we may
have new issue for the following other than HDFS-7337 (which mainly focuses on support of
multiple codecs). Would we have one ?
// TODO: move erasure coding policy to file XAttr (HDFS-7337)
6. Better rename {{BlockCodecInfo}} since the system has many codec concepts.
7. In the test testMissingStripedBlock:
+    assertTrue("There should be 4 outstanding EC tasks", cnt > 0);
1) Can 4 be a variable like {{expectedRecoveryTasks}} to be calculated with constant value
2) Then, have {{cnt == expectedRecoveryTasks}} ?
8. Regarding below, I haven't better idea, and it looks good to me. Maybe we have some comments
there for future optimization ? 
bq.how to efficiently get the indices of missing blocks. Maybe something like the below?

> Erasure coding: distribute recovery work for striped blocks to DataNode
> -----------------------------------------------------------------------
>                 Key: HDFS-7369
>                 URL: https://issues.apache.org/jira/browse/HDFS-7369
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-7369-000-part1.patch, HDFS-7369-000-part2.patch, HDFS-7369-001.patch,
> This JIRA updates NameNode to handle background / offline recovery of erasure coded blocks.
It includes 2 parts:
> # Extend {{UnderReplicatedBlocks}} to recognize EC blocks and insert them to appropriate
priority levels. 
> # Update {{ReplicationMonitor}} to distinguish block codec tasks and send a new DataNode

This message was sent by Atlassian JIRA

View raw message