hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Thu, 11 Aug 2016 22:55:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418082#comment-15418082
] 

Zhe Zhang commented on HDFS-7859:
---------------------------------

A few more comments:
# I think we should be more clear in code and comment that we only persist the user-added
(non-builtin) policies
{code}
+  public void saveErasureCodingPolices(DataOutputStream out, String sdPath)
+      throws IOException {
...
+    out.writeInt(activePoliciesByName.size());

+  public PersistState saveState() throws IOException {
+    List<ErasureCodingPolicyProto> ecPolicies = Lists
+        .newArrayListWithCapacity(activePoliciesByName.size());
+    List<ErasureCodingPolicy> hardCodingECPolicies =
+        Arrays.asList(SYS_POLICIES);
{code}
In above code, we should probably only use the number of user-added policies.
# In {{loadErasureCodingPolicies}} and {{loadState}}, we should probably also check and WARN
if a built-in ({{SYS_POLICIES}}) policy is loaded.
# {{FSDirErasureCodingOp#addErasureCodingPolicy}} can be renamed as {{unprotectedAddErasureCodingPolicy}}
to be clearer about locking
# In below code, maybe a clearer approach is to use {{FSDirErasureCodingOp#unprotectedAddErasureCodingPolicy}}
{code}
+      fsNamesys.getErasureCodingPolicyManager()
+          .addErasureCodingPolicy(addOp.getEcPolicy());
{code}
# The asserts can be replaced by a Precondition check. Maybe we should add the check in {{ErasureCodingManager#addErasureCodingPolicy}}
too
{code}
+    public AddErasureCodingPolicyOp setErasureCodingPolicy(
+        ErasureCodingPolicy ecPolicy) {
+      assert(ecPolicy.getName() != null);
+      assert(ecPolicy.getSchema() != null);
+      assert(ecPolicy.getCellSize() != 0);
{code}
# {{AddErasureCodingPolicyOp#toString}} should rely on {{ErasureCodingPolicy#toString}} to
construct a String for the policy
# Below change is not necessary:
{code}
-      fsn.getCacheManager().loadState(
-          new CacheManager.PersistState(s, pools, directives));
+      fsn.getCacheManager()
+          .loadState(new CacheManager.PersistState(s, pools, directives));
{code}

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Xinwei Qin 
>              Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do
>         Attachments: HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch,
HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message