hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Wed, 08 Feb 2017 02:04:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857239#comment-15857239

Andrew Wang commented on HDFS-7859:

Thanks for the comments Zhe, inline,

bq. Other than "ID/name that we already use, or an ID/name we might want to hardcode later",
what other validations do you have in mind?

I think it's problematic if user-defined policies are in the same id-space/namespace as system-defined
policies. Separating these spaces would eliminate the possibility of overlap and the need
for validation.

bq.  If we do decide to add pluggable EC policies in 3.0 GA, can we add an on-off config option
for the entire pluggable logic and default to off?

Yea, I think that works. IMO it's unlikely we get to pluggable policies at all in 3.0. The
goal as I understand it was to support something like LRC as a pluggable policy, but that
requires hooking into blockplacement and recovery. We haven't 100% finished those yet even
for our built-in policies, and making these interfaces pluggable will be tough.

So, I'd like to co-opt this JIRA to add APIs for defining allowed policies as well as the
default policy for a cluster. We'll need new shell commands for doing this too.

[~xinwei] / [~surendrasingh] are you interested in working on this? If not, I can pick it

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Xinwei Qin 
>            Priority: Blocker
>              Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do
>         Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch,
HDFS-7859.006.patch, HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, HDFS-7859-HDFS-7285.002.patch,
HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message