hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
Date Tue, 13 Sep 2016 22:51:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488709#comment-15488709
] 

Kai Zheng commented on MAPREDUCE-6774:
--------------------------------------

Thanks Sammi for the update! It looks good and only some minors now.

1. Could you define constants for: {{test.io.erasure.code.policy}}, {{test.io.block.storage.policy}}?
2. Would like to see minor refinements for {{checkErasureCodePolicy}}. For example, having
some line breaks, avoiding the {{else}}.

{code}
+  private boolean checkErasureCodePolicy(String erasureCodePolicyName,
+      FileSystem fs, TestType testType) throws IOException {
+    Collection<ErasureCodingPolicy> list =
+        ((DistributedFileSystem) fs).getAllErasureCodingPolicies();
+    boolean isValid = false;
+    int i = 0;
+    for (ErasureCodingPolicy ec : list) {
+      if (erasureCodePolicyName.equals(ec.getName())) {
+        isValid = true;
+        break;
+      }
+    }
+    if (!isValid) {
+      System.out.println("Invalid erasure code policy: " +
+          erasureCodePolicyName);
+      System.out.println("Current supported erasure code policy list: ");
+      for (ErasureCodingPolicy ec : list) {
+        System.out.println(ec.getName());
+      }
+      return false;
+    } else {
+      if (testType == TestType.TEST_TYPE_APPEND ||
+          testType == TestType.TEST_TYPE_TRUNCATE) {
+        System.out.println("So far append or truncate operation" +
+            " with erasureCodePolicy enabled is not supported");
+        return false;
+      }
+    }
+    config.set("test.io.erasure.code.policy", erasureCodePolicyName);
+    LOG.info("erasureCodePolicy = " + erasureCodePolicyName);
+    return true;
+  }
{code}

> Add support for HDFS erasure code policy to TestDFSIO
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-6774
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: SammiChen
>            Assignee: SammiChen
>         Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch, MAPREDUCE-6774-v3.patch,
MAPREDUCE-6774-v4.patch
>
>
> HDFS erasure code policy allows user to store directory and file to predefined erasure
code policies. Currently only 3x replication is supported in TestDFSIO implementation. This
is going to add an new option to enable tests of files with erasure code policy enabled. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message