hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
Date Wed, 10 May 2017 21:17:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005454#comment-16005454
] 

Kai Zheng commented on HDFS-7337:
---------------------------------

Thanks [~Sammi] for moving this on. 

The PluggableErasureCodec-v4 design reflects all the latest discussions (mainly with [~andrew.wang],
[~Sammi] and [~jojochuang]) and related work. Important ones are:
HADOOP-13200 Implement customizable and configurable erasure coders;
HDFS-11314 Enforce set of enabled EC policies on the NameNode;
HDFS-11605 Allow user to customize new erasure code policies.

To close the gap targeting for the upcoming {{3.0 ALPHA3}}  release we're also tightly working
on these:
HDFS-11794 Add ec sub command -listCodec to show currently supported ec codecs
HDFS-11606 Add CLI cmd to remove an erasure code policy
HDFS-7859 Erasure Coding: Persist erasure coding policies in NameNode

As we're close and will make the changes into NameNode fsimage/editlog in HDFS-7859, would
it be great if more folks take a look at the latest design doc and see if any concerns. Particularly,
could I ping [~szetszwo], [~jingzhao], [~zhz] and [~vinayrpet]. Thanks for your time!

> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
>                 Key: HDFS-7337
>                 URL: https://issues.apache.org/jira/browse/HDFS-7337
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Zhe Zhang
>            Priority: Critical
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, PluggableErasureCodec
v4.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs
via pluggable approach. It allows to define and configure multiple codec schemas with different
coding algorithms and parameters. The resultant codec schemas can be utilized and specified
via command tool for different file folders. While design and implement such pluggable framework,
it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework
is useful and workable. Separate JIRA could be opened for the RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation to make
concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level
stuffs that interact with configuration, schema and etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message