hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Wed, 30 Aug 2017 00:40:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146400#comment-16146400
] 

Lei (Eddy) Xu commented on HDFS-7859:
-------------------------------------

Hi, [~Sammi]

Thanks for the latest patch. I have some questions regarding the latest patch.

{code}
private byte state;   // 0x01 - enabled, 0x02 - deleted
{code}

Can we use enum here to be more explicitly?

{code}
message ErasureCodingPolicyManagerSection {
351	    // number of enabled EC policies
352	    required uint32 numEnabledPolicies = 1;
353	    // number of not enabled policies, including system and user defined policy
354	    required uint32 numOtherPolicies = 2;
355	    // repeated ErasureCodingPolicyProto enabledPolicies;
356	    // repeated ErasureCodingPolicyProto otherPolicies;
357	}
{code}
Are only the number of policies persisted? It looks off to me.  It will depends on the order
of system pre-defined policies. So when the pluggable EC policy being merged, would that impact
the correctness of loading / saving fsimage? It _might_ also make upgrade / downgrade difficult.


{code}
public HashSet<String> getEcPoliciesOnDir() {
239	      return ecPoliciesOnDir;
240	    }
241	
242	    public HashSet<Byte> getEcPoliciesOnFile() {
243	      return ecPoliciesOnFile;
244	    }
245	
246	    public void setCountEcPolicies(Boolean enabled) {
247	      countEcPolicies = enabled;
248	    }
249	
{code}

What are the consumers of these functions? lso 

{code}
public static final class PersistState {
246	    public final ErasureCodingPolicyManagerSection section;
247	    public final List<ErasureCodingPolicy> enabledPolicies;
248	    public final List<ErasureCodingPolicy> otherPolicies;
{code}

Can these fields be {{private final}}?

{code}
// FSImage.java
import org.apache.hadoop.hdfs.protocol.IllegalECPolicyException;
{code}

Seems not necessary? also in {{FSImageFormat.java}}

{code}
private INode loadINode(INodeSection.INode n) throws IOException 
{code}
Is this {{IOE}} necessary?


{code}
// TestFSEditLogLoader.java
      cluster.shutdown();
754	      cluster = null;
755	    } finally {
756	      if (cluster != null) {
757	        cluster.shutdown();
758	      }
759	    }
{code}

In general, you can use {{try...resource}} for {{MiniDFSCluster}} .


A more general question to [~Sammi], do you think whether it is possible to get this in beta1?

Thanks!

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: SammiChen
>         Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch,
HDFS-7859.006.patch, HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, HDFS-7859.010.patch,
HDFS-7859.011.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message