hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
Date Fri, 15 May 2015 23:27:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546370#comment-14546370
] 

Craig Welch commented on YARN-3626:
-----------------------------------

bq. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar
into YARN.

That's why I took the config based approach, which apparently is invalid... but it also worked,
which is quite confusing.  I'm going to go back and validate our reasoning for believing it
shoudn't.

bq. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar
into YARN.

I suppose we can if it would work.  It needs to be something which can be propagated from
Oozie, which adds additional complexity.  Ideally, we need something the MrApps can set based
on the presence of the mapred configuration so that it propagates through.  Do we have an
example of this being done elsewhere?

> On Windows localized resources are not moved to the front of the classpath when they
should be
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3626
>                 URL: https://issues.apache.org/jira/browse/YARN-3626
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>         Environment: Windows
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>             Fix For: 2.7.1
>
>         Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch
>
>
> In response to the mapreduce.job.user.classpath.first setting the classpath is ordered
differently so that localized resources will appear before system classpath resources when
tasks execute.  On Windows this does not work because the localized resources are not linked
into their final location when the classpath jar is created.  To compensate for that localized
jar resources are added directly to the classpath generated for the jar rather than being
discovered from the localized directories.  Unfortunately, they are always appended to the
classpath, and so are never preferred over system resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message