hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
Date Fri, 28 Mar 2014 03:15:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950315#comment-13950315
] 

Sangjin Lee commented on MAPREDUCE-5813:
----------------------------------------

The patch looks good to me. The only nit is to add the right job conf before Limits.init()
so that Limits.init() can use the right job conf.

It's nearly the same fix, but how about 

{code}
final JobConf job = new JobConf(MRJobConfig.JOB_CONF_FILE);
Limits.init(job);
...
{code}

> YarnChild does not load job.xml with mapreduce.job.classloader=true 
> --------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5813
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, task
>    Affects Versions: 2.3.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>            Priority: Blocker
>         Attachments: MAPREDUCE-5813.v01.patch
>
>
> {{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}} that relies
on class loading. When {{mapreduce.job.classloader=true}} the job-speicific part of the class
path is separated from {{CLASSPATH}} into {{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible
for the default class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
> This problem is easily avoided by using  {{JobConf.addResource(Path)}} to read the localized
{{job.xml}} without relying on class loading.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message