hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5814) fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
Date Thu, 27 Mar 2014 18:44:25 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949774#comment-13949774

Gera Shegalov commented on MAPREDUCE-5814:

Sangjin, I was thinking of this... but *-site.xml unlike job.xml are not meant to be shipped,
they are meant to be installed.  default.xml is packaged with hadoop, and can therefore make
it into a fat jar. It's a good idea as a defensive measure. The only reason that prevented
me from doing this is that the list of such files can be quite long, and include all kinds
of log4j and other properties. But they can be legally shipped too. So far I have settled
for the least requirement. Looking forward to the pros and cons.

> fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
> ------------------------------------------------------------------------
>                 Key: MAPREDUCE-5814
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5814
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.3.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5814.v01.patch
> We faced a failure when a job.jar compiled against 0.20+ hadoop artifacts had to run
with {{mapreduce.job.classloader=true}} because it needed a more recent guava as a dependency.
The job failed because the cluster's {{*-default.xml}} files were overshadowed by the ones
in the fat jar. We propose to treat these default config files like the system packages {{org.apache.hadoop.}}
to avoid a counterintuitivie behavior as if we had {{mapreduce.job.user.classpath.first}}

This message was sent by Atlassian JIRA

View raw message