hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prabhu Joseph (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-12512) hadoop fs -ls / fails when we use Custom -Dhadoop.root.logger
Date Tue, 27 Oct 2015 08:25:27 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prabhu Joseph updated HADOOP-12512:
-----------------------------------
    Attachment: HADOOP-12512.001.patch

Patch

> hadoop fs -ls / fails when we use Custom -Dhadoop.root.logger 
> --------------------------------------------------------------
>
>                 Key: HADOOP-12512
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12512
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.0
>            Reporter: Prabhu Joseph
>         Attachments: HADOOP-12512.001.patch
>
>
> hadoop fs -ls / fails with below error when we use Custom -Dhadoop.root.logger that creates
Configuration object and adds defaultResource custom-conf.xml with quiet = false.
> custom-conf.xml is optional configuration.
> Exception in thread "main" java.lang.RuntimeException: custom-conf.xml not found
>         at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2612)
>         at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
>         at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
>         at org.apache.hadoop.conf.Configuration.set(Configuration.java:1156)
>         at org.apache.hadoop.conf.Configuration.set(Configuration.java:1128)
>         at
> org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1464)
>         at
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
>         at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
>         at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
>         at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> 		
> 		
> ISSUE:
> ######
> There is a Logic issue in Configuration class and defaultResources list.
> Configuration is shared by classes. Configuration has a shared list of default resources
added by the classes.
> If class A wants x,y,z resources and says all are optional using quiet = false, Configuration
loads if they are present else skips and adds them to list.
> Now shared list i.e defaultResources has x,y,z
> Now if class B wants x resource and says it as mandatory , loadResources scans the entire
list and treats them mandatory. So during scan of y, it will fail.
> where A is Custom Class
> and B is FsShell
> FsShell checks for custom-conf.xml and treats it mandatory and fails.
> 1. The mandatory/optional has to be resource wise. [OR]
> 2. defaultResources should not be shared. 
> Both of them looks complex. And simple fix is the below.
> 1. when loadResource skips initially if resource not found, it has to remove the entry
from defaultResource list as well. There is no use in having a resource 
> which is not at classpath in the list.
> CODE CHANGE:  class org.apache.hadoop.conf.Configuration
> ############
> private Resource loadResource(Properties properties, Resource wrapper, boolean
> quiet) {}
>  ......
>  if (root == null) {
>         if (doc == null) {
>                   if (quiet) {
>         defaultResources.remove(resource);  // FIX: During skip, remove
> Resource from shared list 
>                     return null;
>                   }
>         throw new RuntimeException(resource + " not found");
>         }
>      root = doc.getDocumentElement();
>  }
>  ........
> Tested after code fix, runs successfully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message