hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Masatake Iwasaki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7056) NN unintentionally reads mapred-site.xml
Date Tue, 22 Apr 2014 07:29:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976484#comment-13976484
] 

Masatake Iwasaki commented on HADOOP-7056:
------------------------------------------

I saw this on the DataNode of recent trunk too. The cause was the same.
{code}
    [1] org.apache.hadoop.util.ReflectionUtils.setJobConf (ReflectionUtils.java:91)      
                                                                                         
             
    [2] org.apache.hadoop.util.ReflectionUtils.setConf (ReflectionUtils.java:75)         
                                                                                         
             
    [3] org.apache.hadoop.util.ReflectionUtils.newInstance (ReflectionUtils.java:133)    
                                                                                         
             
    [4] org.apache.hadoop.security.Groups.<init> (Groups.java:64)                  
                                                                                         
                   
    [5] org.apache.hadoop.security.Groups.getUserToGroupsMappingService (Groups.java:240)
                                                                                         
             
    [6] org.apache.hadoop.security.UserGroupInformation.initialize (UserGroupInformation.java:266)
                                                                                         
    
    [7] org.apache.hadoop.security.UserGroupInformation.setConfiguration (UserGroupInformation.java:294)
                                                                                        
    [8] org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode (DataNode.java:1,770)
                                                                                         
     
    [9] org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode (DataNode.java:1,813)
                                                                                         
          
    [10] org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain (DataNode.java:1,990)
                                                                                         
             
    [11] org.apache.hadoop.hdfs.server.datanode.DataNode.main (DataNode.java:2,014)      
                                                                                         
             
{code}

> NN unintentionally reads mapred-site.xml
> ----------------------------------------
>
>                 Key: HADOOP-7056
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7056
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Priority: Critical
>
> It seems that {{mapred-site.xml}} is now unintentionally being read by the NN, and probably
all the daemons.
> This is because the {{Groups}} constructor calls {{ReflectionUtils.newInstance(...)}},
which in turn calls {{ReflectionUtils.setConf(...)}}, which calls {{ReflectionUtils.setJobConf(...)}},
which creates an instance of {{o.a.h.mapred.JobConf}}, which calls {{o.a.h.mapreduce.util.ConfigUtil.loadResources(...)}},
which reads in {{mapred-site.xml}} and {{mapred-default.xml}}. Phew!
> I suspect the appropriate solution is for {{ReflectionUtils.setConf(...)}} to be changed
to not call (the mapred-specific) {{ReflectionUtils.setJobConf(...)}}, but there's a comment
in there which says it's being left in place for backward compatibility. Can somebody with
more knowledge of that situation please comment?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message