hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3626) On Windows localized resources are not moved to the front of the classpath when they should be
Date Fri, 15 May 2015 23:24:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546365#comment-14546365
] 

Chris Nauroth commented on YARN-3626:
-------------------------------------

I don't fully understand the objection to the former patch that had been committed.

bq. The new configuration added is supposed to be per app, but it is now a server side configuration.

There was a new YARN configuration property for triggering this behavior, but the MR application
would toggle on that YARN property only if the MR job submission had {{MAPREDUCE_JOB_USER_CLASSPATH_FIRST}}
on.  From {{MRApps}}:

{code}
    boolean userClassesTakesPrecedence = 
      conf.getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false);

    if (userClassesTakesPrecedence) {
      conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE,
        "true");
    }
{code}

I thought this implemented "per app" behavior, because it could vary between MR app submission
instances.  It would not be a requirement to put {{YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE}}
into the server configs and have the client and server share configs.

Is there a detail I'm missing?

> On Windows localized resources are not moved to the front of the classpath when they
should be
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3626
>                 URL: https://issues.apache.org/jira/browse/YARN-3626
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>         Environment: Windows
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>             Fix For: 2.7.1
>
>         Attachments: YARN-3626.0.patch, YARN-3626.4.patch, YARN-3626.6.patch, YARN-3626.9.patch
>
>
> In response to the mapreduce.job.user.classpath.first setting the classpath is ordered
differently so that localized resources will appear before system classpath resources when
tasks execute.  On Windows this does not work because the localized resources are not linked
into their final location when the classpath jar is created.  To compensate for that localized
jar resources are added directly to the classpath generated for the jar rather than being
discovered from the localized directories.  Unfortunately, they are always appended to the
classpath, and so are never preferred over system resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message