hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2929) Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows support
Date Fri, 12 Dec 2014 02:05:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243590#comment-14243590

Tsuyoshi OZAWA commented on YARN-2929:

Sorry for the shortage of explanations and thanks for your clarification.

I am assuming here that a path originating from a shell script would get passed into a Spark
Java process, where Spark code would have an opportunity to use the Path class like I described.

In fact, this problem happens when launching AM - before launching JVM of the AM.
For instance, AM need classpath to launch. Sometimes classpath includes subdirectory with
path separators. This is a case. The path separators cannot be parsed on the OS and fails
to find jar.

// JVM option in launch-container.sh
-classpath=local/lib/hadoop # cannot be parsed on Windows

For the case, following path is converted into platform depend paths:
// JVM option in launch-container.sh
-classpath=local<FPS>lib<FPS>hadoop # can be converted into platform depend paths
by expandEnvironments

By minimizing token replacement rules like this, we'd reduce the number of special cases that
YARN application writers would need to consider.

Yes, I agree with you. If we don't need it, we shouldn't include the modifications like this.

> Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows support
> ------------------------------------------------------------------------------------
>                 Key: YARN-2929
>                 URL: https://issues.apache.org/jira/browse/YARN-2929
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2929.001.patch
> Some frameworks like Spark is tackling to run jobs on Windows(SPARK-1825). For better
multiple platform support, we should introduce ApplicationConstants.FILE_PATH_SEPARATOR for
making filepath platform-independent.

This message was sent by Atlassian JIRA

View raw message