hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bing Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6015) Make MR ApplicationMaster disable loading user's jars firstly
Date Mon, 04 Aug 2014 10:41:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084528#comment-14084528

Bing Jiang commented on MAPREDUCE-6015:

[~sjlee0], Thanks very much.  Agree with you at the point of 'mapreduce.job.classloader',
it can tackle lots of issue of class conflicts. However, if the user's jar contains some conflict
classes of launching AM. It seems that there is no way to finish using that property. e.g.,
jetty-6.1.26 (hadoop system jars) is conflicting with jetty-6.1.11.jar (users jar). AM failed
to launch.
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
    at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:813)
    at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)

And user logic code is dependent upon jetty-6.1.11.jar, so it cannot disable mapreduce.user.classpath.first.
At this time, it gives me a impression to treat AM and normal Tasks separately.  

> Make MR ApplicationMaster disable loading user's jars firstly 
> --------------------------------------------------------------
>                 Key: MAPREDUCE-6015
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6015
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster
>    Affects Versions: 2.4.1
>            Reporter: Bing Jiang
> In most cases, we want to use -Dmapreduce.user.classpath.first=true to pick user's jars
ahead of hadoop system's jars, which can make tasks run based upon the customized environment
under the circumstance that hadoop system default library contains the different version of
dependent jars.
> However, using -Dmapreduce.user.classpath.first=true will cause ApplicationMaster failure
to launch due to conflicting classes.
> In most cases, if users do not customize the ApplicationMaster for MapReduce framework,
I believe we can treat MRAppMaster different with MapTask/ReduceTask at the point of loading
user's jar in classloader. 
> I believe it can provide  a property of  '-Dmapreduce.am.user.classpath.first=false'
 to disable the feature of loading user's jars firstly. 

This message was sent by Atlassian JIRA

View raw message