hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudharsan Sampath (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would configure anything Configurable twice
Date Wed, 13 Jul 2011 11:32:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064494#comment-13064494

Sudharsan Sampath commented on HADOOP-7425:

Hi Steven,

Yes. You are right. This issue is seen on the trunk too. But to me, configuring twice should
not be an issue but processing the configured property without realizing that the same configuration
has been processed already could be avoided.

Will it be ok if instead of maintaining a List, we maintain a Set of keyspecs in the KeyFieldHelper
and provide an equals method that returns true if its params match exactly as the passed in.
The downside is that if some job wants this kind of double hashing done on the same keyspec
that would be difficult asking for a custom partitioner. But I doubt if there would be such
a scenario.

This would also ensure we get the correct partition and increase performance as we avoid a
hash invoke for every record. 

> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>                 Key: HADOOP-7425
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7425
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.23.0
>            Reporter: steven zhuang
>         Attachments: test.tar
> In  the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable
would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured
twice, it get two KeyDescription and gives out wrong partition number. 
> public static void setConf(Object theObject, Configuration conf) {
>     if (conf != null) {
>       if (theObject instanceof Configurable) {
>         ((Configurable) theObject).setConf(conf);
>       }
>       setJobConf(theObject, conf);
>     }
>   }

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message