Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 15 Dec 2015 07:49:46 +0000 (UTC)
From: "GAO Rui (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12752638.1415083370000.23740.1450165786813@Atlassian.JIRA>
In-Reply-To: <JIRA.12752638.1415083370000@Atlassian.JIRA>
References: <JIRA.12752638.1415083370000@Atlassian.JIRA>
 <JIRA.12752638.1415083370833@arcas>
Subject: [jira] [Commented] (HDFS-7344) [umbrella] Erasure Coding worker and
 support in DataNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057515#comment-15057515 ] 

GAO Rui commented on HDFS-7344:
-------------------------------

[~zhz] Thank you very much for your information. I am trying to draft a design and then implement {{Converter}} which is mentioned in HDFS-7717. I think maybe we should design as {{Converter}} distribute conversation tasks to several {{ErasureCodingWorker}}, then actually these {{ErasureCodingWorker}} implement conversation tasks. Could you share your opinions?  Currently, if put Erasure Coding to product clusters, {{Converter}} should be among the most used functions. Lots of replication files would be converted to EC files. Without {{Converter}}, we can only use distcp. {{Converter}} should could be much more efficient, right? 

> [umbrella] Erasure Coding worker and support in DataNode
> --------------------------------------------------------
>
>                 Key: HDFS-7344
>                 URL: https://issues.apache.org/jira/browse/HDFS-7344
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Kai Zheng
>            Assignee: Li Bo
>         Attachments: ECWorker-design-v2.pdf, HDFS ECWorker Design.pdf, hdfs-ec-datanode.0108.zip, hdfs-ec-datanode.0108.zip
>
>
> According to HDFS-7285 and the design, this handles DataNode side extension and related support for Erasure Coding. More specifically, it implements {{ECWorker}}, which reconstructs lost blocks (in striping layout).
> It generally needs to restore BlockGroup and schema information from coding commands from NameNode or other entities, and construct specific coding work to execute. The required block reader, writer, either local or remote, encoder and decoder, will be implemented separately as sub-tasks. 
> This JIRA will track all the linked sub-tasks, and is responsible for general discussions and integration for ECWorker. It won't resolve until all the related tasks are done.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)