hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12395) Support erasure coding policy operations in namenode edit log
Date Mon, 11 Sep 2017 03:28:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160615#comment-16160615

Kai Zheng commented on HDFS-12395:

Thanks [~Sammi] for working on this. Having looked into the work and it looks overall close.

1. Overall, similar to HDFS-7859, this exposes lots of not-so-relevant changes already made
in the patch or we need to make, so please feel free to open new issues to hold such changes
separately. The essential changes for this issue is to *log add/remove/enable/disable erasure
coding policy*. 

2. You local change in {{pom.xml}} and {{editsStored.xml}}.

3. You changes in {{DFSClient}} looks like some bug fix to existing codes.

4. Refactor: {{getEcPolicy}} => {{getErasureCodingPolicy}}; {{AddECPolicyResponse}} =>

5. Don't quite like the way to sort the map by creating a tree map. And also, could we improve
{{ECSchema}} to ensure {{extraOptions}} is sorted already, so we don't need to consider doing
it in every places? If you'd do this, please in separate issue. 
+      // Sort extra options based on key
+      extraOptions = new TreeMap<String, String>(extraOptions);

6. Please use some non-meaningful options for the test purpose to avoid possible confusion,
like "testOption1" or the like.
+    Map<String, String> extraOptions = new HashMap<String, String>();
+    extraOptions.put("padding", "0");
+    extraOptions.put("recycle", "true");

7. Please try to use the same order when dump fields of erasure coding policy.
+  public static void writeErasureCodingPolicy(DataOutputStream out,
+      ErasureCodingPolicy ecPolicy) throws IOException {
+    writeInt(ecPolicy.getCellSize(), out);
+    writeString(ecPolicy.getSchema().getCodecName(), out);
+    writeInt(ecPolicy.getNumDataUnits(), out);
+    writeInt(ecPolicy.getNumParityUnits(), out);
+      XMLUtils.addSaxString(contentHandler, "CODEC", ecPolicy.getCodecName());
+      XMLUtils.addSaxString(contentHandler, "CELLSIZE",
+          Integer.toString(ecPolicy.getCellSize()));
+      XMLUtils.addSaxString(contentHandler, "DATAUNITS",
+          Integer.toString(ecPolicy.getNumDataUnits()));
+      XMLUtils.addSaxString(contentHandler, "PARITYUNITS",
+          Integer.toString(ecPolicy.getNumParityUnits()));

8. Not sure why we need to catch and convert the exception here, but not in other places.
Better to pass the {{e}} instead of its {e.getMessage()} when convert to new IOException.

+      AddErasureCodingPolicyOp addOp = (AddErasureCodingPolicyOp) op;
+      try {
+        fsNamesys.getErasureCodingPolicyManager().addPolicy(
+            addOp.getEcPolicy());
+      } catch (HadoopIllegalArgumentException e) {
+        throw new IOException("Add erasure coding policy failed for:" +
+            e.getMessage());
+      }

> Support erasure coding policy operations in namenode edit log
> -------------------------------------------------------------
>                 Key: HDFS-12395
>                 URL: https://issues.apache.org/jira/browse/HDFS-12395
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding
>            Reporter: SammiChen
>            Assignee: SammiChen
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: editsStored, HDFS-12395.001.patch, HDFS-12395.002.patch
> Support add, remove, disable, enable erasure coding policy operation in edit log. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message