hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Eagles (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12490) Add default resources to Configuration which are not inside the class path
Date Fri, 30 Nov 2018 16:20:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704969#comment-16704969
] 

Jonathan Eagles commented on HADOOP-12490:
------------------------------------------

[~mjacob_otto], this will have to be done differently since jdk9+ doesn't support this capability
by default. TEZ-3860

> Add default resources to Configuration which are not inside the class path
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-12490
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12490
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 2.7.1
>            Reporter: Matthias Jacob
>            Assignee: Matthias Jacob
>            Priority: Major
>         Attachments: HADOOP-12490.001.patch
>
>
> *Problem*
> It is currently not possible to add default resources to {{org.apache.hadoop.conf.Configuration}}
that are not included in the class path.
> We need to add resources to {{org.apache.hadoop.conf.Configuration}} that are not included
in the class path. Currently, the only way to do this is to use {{addResource(URL)}} or {{addResource(Path)}}
which both are instance methods. However, a library we work with creates Configuration instances
internally that we cannot access. So we need to add them as default resources in order to
make them available to all subsequent Configuration instances (and those existing instances
where {{loadDefaults}} equals {{true}}, of course).
> *Solution*
> The proposed solution is to add the following static methods:
> {code}
>   public static synchronized void addDefaultResource(URL url);
>   public static synchronized void addDefaultResource(Path file);
> {code}
> Please see the attached patch for details.
> *Example*
> In order to initialize the Configuration correctly, what we need to do currently is as
follows:
> {code}
> // Set resource paths for the class loader
> File ffile = new File("/path/to/conf/dir");
> URL url = ffile.toURI().toURL();
> URLClassLoader systemClassLoader = (URLClassLoader) getClass().getClassLoader();
> Class<URLClassLoader> classLoaderClass = URLClassLoader.class; 
> Method method = classLoaderClass.getDeclaredMethod("addURL", new Class[]{URL.class});

> method.setAccessible(true); 
> method.invoke(systemClassLoader, new Object[]{url}); 
> //-----------------------------
> // Read cluster config
> for (File f : ffile.listFiles()) {
>       if(f.getName().toLowerCase().endsWith(".xml")) {
>             Configuration.addDefaultResource(f.getName());
>       }
> }
> {code}
> As you can see, we currently need to manipulate the class path through reflection which
is kind of dirty.
> With the proposed solution, the same (adding default resources from outside the class
path) can be achieved doing the following:
> {code}
> // Set resource paths for the class loader
> File ffile = new File("/path/to/conf/dir");
> //-----------------------------
> // Read cluster config
> for (File f : ffile.listFiles()) {
>       if(f.getName().toLowerCase().endsWith(".xml")) {
>             Configuration.addDefaultResource(new Path(f.getAbsolutePath()));
>       }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message