hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Yao (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
Date Thu, 27 Apr 2017 03:38:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985929#comment-15985929
] 

Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:37 AM:
-----------------------------------------------------------

I have double checked the code and don't think it's related to this patch either, as CodecUtil
is never used in this class.
By the way, I examined some other bugs when checking all the usages of CodecUtil. As the configuration
of erasure code has been changed, some configurations should be changed accordingly in unit
tests. The bugs lie in the codes like the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
    conf.set(
      CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
      NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no error displayed.
I will correct them immediately.


was (Author: timmyyao):
I have double checked the code and don't think it's related to this patch either, as CodecUtil
is never used in this class.
By the way, I examined some other bugs when checking all the usages of CodecUtil. As the configuration
of erasure code has been changed, some configurations should be changed accordingly. The bugs
lie in the codes like the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
    conf.set(
      CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
      NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no error displayed.
I will correct them immediately.

> Seeking a better approach allowing to customize and configure erasure coders
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-13200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13200
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Tim Yao
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch, HADOOP-13200.04.patch,
HADOOP-13200.05.patch, HADOOP-13200.06.patch, HADOOP-13200.07.patch, HADOOP-13200.08.patch,
HADOOP-13200.09.patch, HADOOP-13200.10.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may be some
better approach allowing to customize and configure erasure coders than the current having
raw coder factory, as [~cmccabe] suggested. Will copy the relevant comments here to continue
the discussion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message