hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
Date Thu, 23 Mar 2017 06:50:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937831#comment-15937831

Kai Zheng commented on HDFS-7337:

Thanks Andrew!

bq. It might be simpler for admins if we still do this over an RPC interface. Rather than
specifying all the ECSchema info as arguments, the CLI tool can take the XML file as input.
The CLI tool can also perform basic validation, and prompt the user when doing possibly destructive
operations like removing a schema.
It's a great new suggestion and it sounds much better as you said. We use an XML file to define
codecs, schemas and policies, and then have CLI parse, validate, send over RPC and load them
into NameNode side. One thing left, do we need an XML sample file put in the configuration
folder for admins to reference? 

Do you think we should allow removing of schema/policy by this XML means? IMO, the XML file
is only for new entries. Extra CLI command could be provided to do removal. When do removal,
codec/schema/policy name would be used to distinguish and reference the entry to remove? No
update is supported, since admins can remove and then add.

Glad we're much close now. Hope we can revive the work soon.

> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>                 Key: HDFS-7337
>                 URL: https://issues.apache.org/jira/browse/HDFS-7337
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Zhe Zhang
>            Assignee: Kai Zheng
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf
> According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs
via pluggable approach. It allows to define and configure multiple codec schemas with different
coding algorithms and parameters. The resultant codec schemas can be utilized and specified
via command tool for different file folders. While design and implement such pluggable framework,
it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework
is useful and workable. Separate JIRA could be opened for the RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation to make
concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level
stuffs that interact with configuration, schema and etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message