hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Thu, 07 Sep 2017 12:28:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156877#comment-16156877
] 

Kai Zheng commented on HDFS-7859:
---------------------------------

Thanks Sammi for the update. The patch seems in much better form.

1. Do you have any issue to track the TODO work? Will you plan to do it shortly for BETA1?
{code}
+    /*
+     * TODO check if there is any file or directory using this policy.
+     * If no, remove this erasure coding policy from system permanently.
+     * */
{code}

2. Hmm, I thought "dfs.namenode.ec.policies.enabled" should have been removed when adding
the API to enable/disable policy. Please checkout the relevant discussions and issues to ensure
this if it hasn't happened already.
bq. HDFS configuration file means "hdfs-site.xml". For example, user can define enabled policies
through property "dfs.namenode.ec.policies.enabled"

Copy the relevant codes again:
{code}
+    for (ErasureCodingPolicy p : getPolicies()) {
+      if (p.getName().equals(policyName) ||
+          (p.getSchema().equals(policy.getSchema()) &&
+          p.getCellSize() == policy.getCellSize())) {
+        // If the same policy loaded from fsImage override policy loaded based
+        // on HDFS configuration file
+        LOG.info("The erasure coding policy name " + policy + " loaded from " +
+            "fsImage overrides the one loaded according to HDFS hdfs-site.xml");
+      }
+    }
{code}

3. Again, could we make the change: ErasureCodingPolicyManagerSection => ErasureCodingSection.
Also check related names like loadErasureCodingPolicyManagerSection, saveErasureCodingPolicyManagerSection.

4. The assert message would be "The erasure coding policy saved into and loaded from fsimage
was bad" to make more sense. 
{code}
+      assertEquals("The erasure coding policy is not found",
+          targetPolicy, ecPolicy);
{code}

5. Here or elsewhere, please ensure no policy can be DISABLED/REMOVED if it's used by files,
with necessary tests.

Thanks!

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: SammiChen
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch,
HDFS-7859.006.patch, HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, HDFS-7859.010.patch,
HDFS-7859.011.patch, HDFS-7859.012.patch, HDFS-7859.013.patch, HDFS-7859.014.patch, HDFS-7859.015.patch,
HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message