hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
Date Thu, 31 Aug 2017 00:08:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148241#comment-16148241

Kai Zheng commented on HDFS-7859:

bq. Are only the number of policies persisted? It looks off to me. It will depends on the
order of system pre-defined policies. So when the pluggable EC policy being merged, would
that impact the correctness of loading / saving fsimage? It might also make upgrade / downgrade
I agree with Eddy on this and have the same concern. We need to persist all the system policies
and user defined policies including their info (name, id, cell size and EC schema) along with
their status (lenabled/disabled, removed). We need to ensure all the persisted info can be
used to recover/export/import/convert data and do the upgrading/downgrading stuffs.

bq. Lei (Eddy) Xu mentioned upgrade and downgrade, it's a good question. Not only user defined
ec policy, but also built-in ec policy will face this issue. The major problem is if a codec
is no longer supported after upgrade or downgrade, how to handle these type of ec policies
in the new cluster, also how to handle the files/directories which used these no long supported
It should be a rare case we need to consider that an EC codec/coder/algorithm will not be
supported and removed from the code base. If user adds some pluggable codec but then remove
it from binary, it's their call. So let's not worry about this at this time.

Let's focus on the basic use cases and requirements, and move on not being too overloaded.

> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>                 Key: HDFS-7859
>                 URL: https://issues.apache.org/jira/browse/HDFS-7859
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: SammiChen
>         Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, HDFS-7859.005.patch,
HDFS-7859.006.patch, HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, HDFS-7859.010.patch,
HDFS-7859.011.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we persist EC schemas
in NameNode centrally and reliably, so that EC zones can reference them by name efficiently.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message