hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2929) Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows support
Date Fri, 12 Dec 2014 02:05:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243590#comment-14243590
] 

Tsuyoshi OZAWA commented on YARN-2929:
--------------------------------------

Sorry for the shortage of explanations and thanks for your clarification.

{quote}
I am assuming here that a path originating from a shell script would get passed into a Spark
Java process, where Spark code would have an opportunity to use the Path class like I described.
{quote}

In fact, this problem happens when launching AM - before launching JVM of the AM.
For instance, AM need classpath to launch. Sometimes classpath includes subdirectory with
path separators. This is a case. The path separators cannot be parsed on the OS and fails
to find jar.

{code}
// JVM option in launch-container.sh
-classpath=local/lib/hadoop # cannot be parsed on Windows
{code}

For the case, following path is converted into platform depend paths:
{code}
// JVM option in launch-container.sh
-classpath=local<FPS>lib<FPS>hadoop # can be converted into platform depend paths
by expandEnvironments
{code}

{quote}
By minimizing token replacement rules like this, we'd reduce the number of special cases that
YARN application writers would need to consider.
{quote}

Yes, I agree with you. If we don't need it, we shouldn't include the modifications like this.

> Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows support
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-2929
>                 URL: https://issues.apache.org/jira/browse/YARN-2929
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2929.001.patch
>
>
> Some frameworks like Spark is tackling to run jobs on Windows(SPARK-1825). For better
multiple platform support, we should introduce ApplicationConstants.FILE_PATH_SEPARATOR for
making filepath platform-independent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message