hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
Date Tue, 23 May 2017 02:27:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020574#comment-16020574

Andrew Wang commented on HDFS-7337:

Hi folks, thanks for the discussion,

bq. Is there a way to choose a system-wide default codec? So that after the cluster being
initialized, users and admins can just specify a zone / directory to be "erasure coded", instead
of choosing from several different codes, and each one has its own trade-offs, which require
user / admin to understand?

We had a system default policy originally, but then moved away from it. I'm open to bringing
it back if we believe that there's typically only one policy in a cluster. I think this is
likely true.

bq. My concern is that, even if the admin is able to add policy via the API dynamically, it
still requires the admin to reboot NN, or ssh into NN / change conf files and reload NN confs,
to enable the policy? It makes the workflow complicated.

Yea, this is true. I can envision how this would work with just CLI commands: add/remove/enable/disable.
I don't know how we'd do this with just config, since we want the safety of persisting things
in the fsimage.

So, shall we do it all via API?

> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>                 Key: HDFS-7337
>                 URL: https://issues.apache.org/jira/browse/HDFS-7337
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Zhe Zhang
>            Priority: Critical
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, PluggableErasureCodec
> According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs
via pluggable approach. It allows to define and configure multiple codec schemas with different
coding algorithms and parameters. The resultant codec schemas can be utilized and specified
via command tool for different file folders. While design and implement such pluggable framework,
it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework
is useful and workable. Separate JIRA could be opened for the RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation to make
concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level
stuffs that interact with configuration, schema and etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message