hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12490) Add default resources to Configuration which are not inside the class path
Date Mon, 03 Jul 2017 18:48:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072812#comment-16072812
] 

Hadoop QA commented on HADOOP-12490:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 16s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 24s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 42s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 38s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  4s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 27s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 50s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 47s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 34s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 34s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 38s{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 25 new + 146 unchanged
- 2 fixed = 171 total (was 148) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  4s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 46s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 54s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 24s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 34s{color} | {color:red}
The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 54s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
|   | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-12490 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12767355/HADOOP-12490.001.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux d693c85d0d19 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bf1f599 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12707/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/12707/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12707/testReport/ |
| asflicense | https://builds.apache.org/job/PreCommit-HADOOP-Build/12707/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
|
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12707/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add default resources to Configuration which are not inside the class path
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-12490
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12490
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 2.7.1
>            Reporter: Matthias Jacob
>            Assignee: Matthias Jacob
>         Attachments: HADOOP-12490.001.patch
>
>
> *Problem*
> It is currently not possible to add default resources to {{org.apache.hadoop.conf.Configuration}}
that are not included in the class path.
> We need to add resources to {{org.apache.hadoop.conf.Configuration}} that are not included
in the class path. Currently, the only way to do this is to use {{addResource(URL)}} or {{addResource(Path)}}
which both are instance methods. However, a library we work with creates Configuration instances
internally that we cannot access. So we need to add them as default resources in order to
make them available to all subsequent Configuration instances (and those existing instances
where {{loadDefaults}} equals {{true}}, of course).
> *Solution*
> The proposed solution is to add the following static methods:
> {code}
>   public static synchronized void addDefaultResource(URL url);
>   public static synchronized void addDefaultResource(Path file);
> {code}
> Please see the attached patch for details.
> *Example*
> In order to initialize the Configuration correctly, what we need to do currently is as
follows:
> {code}
> // Set resource paths for the class loader
> File ffile = new File("/path/to/conf/dir");
> URL url = ffile.toURI().toURL();
> URLClassLoader systemClassLoader = (URLClassLoader) getClass().getClassLoader();
> Class<URLClassLoader> classLoaderClass = URLClassLoader.class; 
> Method method = classLoaderClass.getDeclaredMethod("addURL", new Class[]{URL.class});

> method.setAccessible(true); 
> method.invoke(systemClassLoader, new Object[]{url}); 
> //-----------------------------
> // Read cluster config
> for (File f : ffile.listFiles()) {
>       if(f.getName().toLowerCase().endsWith(".xml")) {
>             Configuration.addDefaultResource(f.getName());
>       }
> }
> {code}
> As you can see, we currently need to manipulate the class path through reflection which
is kind of dirty.
> With the proposed solution, the same (adding default resources from outside the class
path) can be achieved doing the following:
> {code}
> // Set resource paths for the class loader
> File ffile = new File("/path/to/conf/dir");
> //-----------------------------
> // Read cluster config
> for (File f : ffile.listFiles()) {
>       if(f.getName().toLowerCase().endsWith(".xml")) {
>             Configuration.addDefaultResource(new Path(f.getAbsolutePath()));
>       }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message