hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Tue, 14 Feb 2017 22:47:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866857#comment-15866857

Andrew Wang commented on HDFS-7859:

Appreciate the comments Kai, inline:

bq. Making the policy parameter optional could be friendly considering they may have no idea
before the list of available policies promoted to them.

Agree, though I think most of the time these policies are going to be set up once by the admin.

To support friendly usage, we could have a client-side configuration similar to "dfs.replication"
that specifies the default EC policy. Or we could have a NN default when no EC policy is specified.
I think this can be added compatibly later though.

bq. If EC policy info get persisted and stay along with data, users might feel more confident
and comfortable to do data validation and transformation in system upgrading.

Right, that's essentially how it works now with the hardcoded policies. I think if/when we
introduce pluggable policies, we should revive this JIRA so we aren't attaching an entire
EC policy to every EC file that uses a pluggable policy, but until then it's not needed.

I went ahead and filed HDFS-11416 for refactoring out the system default policy, thanks again
for the discussion!

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Andrew Wang
>            Priority: Blocker
>              Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do
>         Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch,
HDFS-7859.006.patch, HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, HDFS-7859-HDFS-7285.002.patch,
HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message