hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
Date Mon, 24 Apr 2017 03:29:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980685#comment-15980685

Tim Yao commented on HADOOP-13200:

Thanks [~jojochuang] for your comments.
Given that a DataNode can have thousands of concurrent client connections on a busy cluster,
adding this extra overhead is not a good idea. Plus, this array will not be updated after
initialization. I think we can do a better job than that.
I think this is a quite important reason for the modification of this part of code. I will
improve it using another array of names to avoid the extra overhead.
Still in CodecRegistry constructor. I think you want to continue instead of break if a coder
has conflict. Otherwise you would throw an exception instead of just log an error message.
I am sorry I don't quite get the point. I use break when finding whether the coder name has
conflict. If the same name has been detected, an error message will be printed and loop for
the coder name detection will be ended. This coder factory will then be ignored and the constructor
will still continue to register the remaining coder factories. 

And I will take the other advises and apply a new patch accordingly.

> Seeking a better approach allowing to customize and configure erasure coders
> ----------------------------------------------------------------------------
>                 Key: HADOOP-13200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13200
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Tim Yao
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch, HADOOP-13200.04.patch,
HADOOP-13200.05.patch, HADOOP-13200.06.patch
> This is a follow-on task for HADOOP-13010 as discussed over there. There may be some
better approach allowing to customize and configure erasure coders than the current having
raw coder factory, as [~cmccabe] suggested. Will copy the relevant comments here to continue
the discussion.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message