hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6015) Make MR ApplicationMaster disable loading user's jars firstly
Date Thu, 31 Jul 2014 15:46:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081023#comment-14081023

Sangjin Lee commented on MAPREDUCE-6015:

I understand the nature of the issue, but it also makes me wonder how effective the proposed
solution would be. There are cases like uberized jobs where the proposed config probably cannot
be used. Even in the case of non-uberized jobs, the MR app master loads several user classes
and those may still need user.classpath.first for their operation.

Curious, have you used mapreduce.job.classloader? To me that should be a much more robust
solution, including issues in the MR app master (with the fix that was recently made in MAPREDUCE-5957).

> Make MR ApplicationMaster disable loading user's jars firstly 
> --------------------------------------------------------------
>                 Key: MAPREDUCE-6015
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6015
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster
>    Affects Versions: 2.4.1
>            Reporter: Bing Jiang
> In most cases, we want to use -Dmapreduce.user.classpath.first=true to pick user's jars
ahead of hadoop system's jars, which can make tasks run based upon the customized environment
under the circumstance that hadoop system default library contains the different version of
dependent jars.
> However, using -Dmapreduce.user.classpath.first=true will cause ApplicationMaster failure
to launch due to conflicting classes.
> In most cases, if users do not customize the ApplicationMaster for MapReduce framework,
I believe we can treat MRAppMaster different with MapTask/ReduceTask at the point of loading
user's jar in classloader. 
> I believe it can provide  a property of  '-Dmapreduce.am.user.classpath.first=false'
 to disable the feature of loading user's jars firstly. 

This message was sent by Atlassian JIRA

View raw message