hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
Date Wed, 03 Feb 2016 23:28:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131369#comment-15131369
] 

Hudson commented on HADOOP-12041:
---------------------------------

FAILURE: Integrated in Hadoop-trunk-Commit #9242 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9242/])
HADOOP-12041. Implement another Reed-Solomon coder in pure Java. (zhezhang: rev c89a14a8a4fe58f01f0cba643f2bc203e1a8701e)
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RSRawEncoder2.java
* hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRSRawCoder.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureDecoder.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RSRawDecoder.java
* hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRSRawCoder2.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/DumpUtil.java
* hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRSRawCoderBase.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/RSUtil2.java
* hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoder.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RSRawErasureCoderFactory2.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/RSUtil.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RSRawDecoder2.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/GF256.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/CoderUtil.java


> Implement another Reed-Solomon coder in pure Java
> -------------------------------------------------
>
>                 Key: HADOOP-12041
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12041
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>             Fix For: 3.0.0
>
>         Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch, HADOOP-12041-v3.patch,
HADOOP-12041-v4.patch, HADOOP-12041-v5.patch, HADOOP-12041-v6.patch, HADOOP-12041-v7.patch,
HADOOP-12041-v8.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation have some drawbacks
or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the decode API
(HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete coding algorithms
and matrix, so Java coders and native coders can be easily swapped in/out and transparent
to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure coding is totally
a byte based data system, we don't need to consider other symbol size instead of 256.
> This desires to implement another  RS coder in pure Java, in addition to the existing
{{GaliosField}} from HDFS-RAID. The new Java RS coder will be favored and used by default
to resolve the related issues. The old HDFS-RAID originated coder will still be there for
comparing, and converting old data from HDFS-RAID systems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message