Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7AE43108F4 for ; Thu, 29 Jan 2015 03:14:35 +0000 (UTC) Received: (qmail 77993 invoked by uid 500); 29 Jan 2015 03:14:35 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 77892 invoked by uid 500); 29 Jan 2015 03:14:35 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 77488 invoked by uid 99); 29 Jan 2015 03:14:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jan 2015 03:14:35 +0000 Date: Thu, 29 Jan 2015 03:14:35 +0000 (UTC) From: "Kai Zheng (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296323#comment-14296323 ] Kai Zheng commented on HADOOP-11514: ------------------------------------ Hi [~anu], Thanks for you question. bq. how does it know if the ECBlock (or ECChunk) is valid ? In high level constructs like ECManager, it will check EC blocks are healthy or not (like lost at all due to DataNode failure or corrupt by block checksum detected), if some block is found in question then the erasure codec framework will play in the work and this *RawErasureCoder* will be involved. The bad block will be marked as isErasred being true and the codec/coder will know which block in a ECGroup or BlockGroup is to be recovered, and then ECChunks will be extracted for decoding. > Raw Erasure Coder API for concrete encoding and decoding > -------------------------------------------------------- > > Key: HADOOP-11514 > URL: https://issues.apache.org/jira/browse/HADOOP-11514 > Project: Hadoop Common > Issue Type: Sub-task > Affects Versions: HDFS-EC > Reporter: Kai Zheng > Assignee: Kai Zheng > Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch > > > This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)