hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Tue, 05 Sep 2017 00:53:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153001#comment-16153001
] 

Kai Zheng commented on HDFS-7859:
---------------------------------

Continued.

1. Better to fix the existing text by the way. "does not exists" => "doesn't exist"
{code}
-      throw new IllegalArgumentException("The policy name " +
+      throw new HadoopIllegalArgumentException("The policy name " +
           name + " does not exists");
{code}

2. PROHIBITED => DISABLED
{code}
+    if (!CodecUtil.hasCodec(policy.getCodecName()) ||
+        policy.getCellSize() > maxCellSize) {
+      // If policy is not supported in current system, set the policy state to
+      // PROHIBITED;
+      policy.setState(ErasureCodingPolicyState.DISABLED);
+    }
{code}

3. What did you mean by policies loaded HDFS configuration file? There isn't any such file
to configure and load EC policies. User may provide one in client, but it's forgotten after
used. 
{code}
+    String policyName = policy.getName();
+    for (ErasureCodingPolicy p : getPolicies()) {
+      if (p.getName().equals(policyName) ||
+          (p.getSchema().equals(policy.getSchema()) &&
+          p.getCellSize() == policy.getCellSize())) {
+        // If the same policy loaded from fsImage override policy loaded based
+        // on HDFS configuration file
+        LOG.info("The erasure coding policy name " + policy + " loaded from " +
+            "fsImage override the one loaded according to HDFS " +
+            "configuration file");
+      }
{code}

4. How about ErasureCodingPolicyManagerSection  => ErasureCodingSection?

5. Could we get rid of {{allPolicies}} or avoid repeatedly creating the array in the for loop
from the map?
{code}
+  public synchronized void reloadPolicy(ErasureCodingPolicy policy) {
...
+    allPolicies = policiesByName.values().toArray(new ErasureCodingPolicy[0]);
...
+  }

+  public synchronized void loadState(PersistState state) {
...
+    for (ErasureCodingPolicy p : state.getPolicies()) {
+      reloadPolicy(p);
+    }
+  }
{code}

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: SammiChen
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch,
HDFS-7859.006.patch, HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, HDFS-7859.010.patch,
HDFS-7859.011.patch, HDFS-7859.012.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch,
HDFS-7859-HDFS-7285.003.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message