hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Per Bergland (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1340) MiniYARNCluster generates wrong style directories in YarnConfiguration.NM_LOCAL_DIR, causes tests to fail if path contains space
Date Fri, 25 Oct 2013 13:22:30 GMT

     [ https://issues.apache.org/jira/browse/YARN-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Per Bergland updated YARN-1340:
-------------------------------

    Description: 
We found that our tests based on the ClusterMapReduceTestCase class failed when the jenkins
job contained spaces and were able to reproduce the error by just renaming the project directory
to contain a space character.
The failure happens when validatePaths method in LocalDirsHandlerService tries to interpret
the paths as URLs "new URL(dir)" and this fails.

https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java

The code in the MiniYARNCluster.prepareDirs method needs to be modified to create properly
escaped file://-based URLs instead of raw file paths OR the receiving end in LocalDirsHandlerService
needs to stop interpreting the directories as urls. Since MiniYARNCluster is a test class
I suspect that the former needs to be done.

  was:
We found that our tests based on the ClusterMapReduceTestCase class failed when the jenkins
job contained spaces and were able to reproduce the error by just renaming the project directory
to create a space character.
The failure happens when validatePaths method in LocalDirsHandlerService tries to interpret
the paths as URLs "new URL(dir)" and this fails.

https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java

The code in the MiniYARNCluster.prepareDirs method needs to be modified to create properly
escaped file://-based URLs instead of raw file paths OR the receiving end in LocalDirsHandlerService
needs to stop interpreting the directories as urls. Since MiniYARNCluster is a test class
I suspect that the former needs to be done.


> MiniYARNCluster generates wrong style directories in YarnConfiguration.NM_LOCAL_DIR,
causes tests to fail if path contains space
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-1340
>                 URL: https://issues.apache.org/jira/browse/YARN-1340
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.0.6-alpha
>         Environment: Mac OS X 10.8.5, CentOS 6.3
>            Reporter: Per Bergland
>
> We found that our tests based on the ClusterMapReduceTestCase class failed when the jenkins
job contained spaces and were able to reproduce the error by just renaming the project directory
to contain a space character.
> The failure happens when validatePaths method in LocalDirsHandlerService tries to interpret
the paths as URLs "new URL(dir)" and this fails.
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
> The code in the MiniYARNCluster.prepareDirs method needs to be modified to create properly
escaped file://-based URLs instead of raw file paths OR the receiving end in LocalDirsHandlerService
needs to stop interpreting the directories as urls. Since MiniYARNCluster is a test class
I suspect that the former needs to be done.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message