hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Yao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13200) Implement customizable and configurable erasure coders
Date Tue, 02 May 2017 11:38:04 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tim Yao updated HADOOP-13200:
-----------------------------
    Release Note: 
CodecRegistry uses ServiceLoader to dynamically load all implementations of RawErasureCoderFactory.
In Hadoop 3.0, there are several built-in implementations, and user can also provide self-defined
implementations with the corresponding resource files. 
For each codec, user can configure the order of the implementations with the configuration
keys:
`io.erasurecode.codec.rs.rawcoders` for the default RS codec,
`io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec,
`io.erasurecode.codec.xor.rawcoders` for the XOR codec.
User can also configure self-defined codec with the configuration key like:
`io.erasurecode.codec.self-defined.rawcoders`.
For each codec, Hadoop will use the implementation according to the order configured. If the
former implementation fails, it will fall back to call the latter one. The order is defined
by a list of coder names separated by commas. The names for the built-in implementations are:
`rs_native` and `rs_java` for the default RS codec, of which  the former is a native implementation
which leverages Intel ISA-L library, which is the default implementation and the latter is
the implementation in pure Java,
`rs-legacy_java` for the legacy RS codec, which is the default implementation in pure Java,
`xor_native` and `xor_java` for the XOR codec, of which the former is the Intel ISA-L implementation
which is the default one and the latter in pure Java.


  was:
CodecRegistry uses ServiceLoader to dynamically load all implementations of RawErasureCoderFactory.
In Hadoop 3.0, there are several built-in implementations, and user can also provide self-defined
implementations with the corresponding resource files. 
For each codec, user can configure the order of the implementations with the configuration
keys:
`io.erasurecode.codec.rs.rawcoders` for the default RS codec,
`io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec,
`io.erasurecode.codec.rs.rawcoders` for the XOR codec.
User can also configure self-defined codec with the configuration key like:
`io.erasurecode.codec.self-defined.rawcoders`.
For each codec, Hadoop will use the implementation according to the order configured. If the
former implementation fails, it will fall back to call the latter one. The order is defined
by a list of coder names separated by commas. The names for the built-in implementations are:
`rs_native` and `rs_java` for the default RS codec, of which  the former is a native implementation
which leverages Intel ISA-L library, which is the default implementation and the latter is
the implementation in pure Java,
`rs-legacy_java` for the legacy RS codec, which is the default implementation in pure Java,
`xor_native` and `xor_java` for the XOR codec, of which the former is the Intel ISA-L implementation
which is the default one and the latter in pure Java.



> Implement customizable and configurable erasure coders
> ------------------------------------------------------
>
>                 Key: HADOOP-13200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13200
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Tim Yao
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>             Fix For: 3.0.0-alpha3
>
>         Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch, HADOOP-13200.04.patch,
HADOOP-13200.05.patch, HADOOP-13200.06.patch, HADOOP-13200.07.patch, HADOOP-13200.08.patch,
HADOOP-13200.09.patch, HADOOP-13200.10.patch, HADOOP-13200.11.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may be some
better approach allowing to customize and configure erasure coders than the current having
raw coder factory, as [~cmccabe] suggested. Will copy the relevant comments here to continue
the discussion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message