hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
Date Thu, 06 Apr 2017 01:50:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958160#comment-15958160

Kai Zheng commented on HDFS-11623:

bq. Should I rename the newly split ErasureCodingPolicies to SystemErasureCodingPolicies for
Yeah, that would be more accurate. Thanks!

bq. the ECPManager is responsible for managing the set of enabled policies, which is different
from the full set of system policies.
I got your idea, so in your view the manager is to manage the set of *enabled* policies (from
system ones and user defined ones). In my thought, we need a central place to manage:
* system policies, or the built-in ones;
* user defined policies, input by an CLI command via an XML file;

* enabled policies;
* available policies to choose and enable;
* maybe disabled policies? For those admins want to blacklist but can't just simply remove
as already used by some data.

It can have :
* get a list of all policies;
* get a policy by name or id;
* deal with fsimage/editlog persisting user defined policies, or doing the removal;
* maybe a method like {{isSystemPolicy}} to tell one policy is system one or not.

Maybe server side tests prefer the manager over the new {{SystemErasureCodingPolicies}} since
it will also support test of user defined policies. 

I think it's a good time to discuss what the manager should do, as quite a few issues on-going
may relate to it.

> Move system erasure coding policies into hadoop-hdfs-client
> -----------------------------------------------------------
>                 Key: HDFS-11623
>                 URL: https://issues.apache.org/jira/browse/HDFS-11623
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch
> This is a precursor to HDFS-11565. We need to move the set of system defined EC policies
out of the NameNode's ECPolicyManager into the hdfs-client module so it can be referenced
by the client.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message