hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
Date Mon, 22 May 2017 21:08:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020209#comment-16020209

Lei (Eddy) Xu commented on HDFS-7337:

Hi, [~drankye] and [~Sammi]

Thanks a lot for the reply. The explanation helps a lot.

bq. There are several system wide codecs to use, including RS codec, RS legacy codec and XOR

Is there a way to choose a system-wide *_default_* codec? So that after the cluster being
initialized,  users and admins can just specify a zone / directory to be "erasure coded",
instead of choosing from several different codes, and each one has its own trade-offs, which
require user / admin to understand? 

bq. while it supports to add / remove policies using CLI,  it dose not support to enable /
use the policy via CLI?

My concern is that, even if the admin is able to add policy via the API *dynamically*, it
still requires the admin to reboot NN, or ssh into NN / change conf files and reload NN confs,
to enable the policy? It makes the workflow complicated.  I think using API / CLI and ssh
NN / changing conf files should be two different sets of operations.  If possible, it is more
consistent to do the EC policy management in either one, or both.  The current design is doing
half of the management in each approach. 


> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>                 Key: HDFS-7337
>                 URL: https://issues.apache.org/jira/browse/HDFS-7337
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Zhe Zhang
>            Priority: Critical
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, PluggableErasureCodec
> According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs
via pluggable approach. It allows to define and configure multiple codec schemas with different
coding algorithms and parameters. The resultant codec schemas can be utilized and specified
via command tool for different file folders. While design and implement such pluggable framework,
it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework
is useful and workable. Separate JIRA could be opened for the RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation to make
concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level
stuffs that interact with configuration, schema and etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message