hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
Date Fri, 07 Mar 2014 15:48:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923971#comment-13923971

Jason Lowe commented on MAPREDUCE-5751:

My apologies, I still don't see why jetty needs to be part of the system classes.  If that's
true then one could argue guava, jackson, jersey, avro, etc. could also have the same issue.
 Couldn't those classes do some initialization in the hadoop classloader context then later
some user code triggers some other internal classes of those dependencies to be loaded and
we end up mixing and matching as we did with jetty?  Or am I misunderstanding the issue and
jetty is doing something none of our other dependencies will do?

I want to make sure I understand the criteria for what should be put in the system classes.
 If I'm understanding the issue properly, it seems like a slippery slope from adding jetty
to adding a large chunk of our dependencies which would effectively undermine the whole point
of the app classloader.

> MR app master fails to start in some cases if mapreduce.job.classloader is true
> -------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5751
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: mapreduce-5751.patch
> If mapreduce.job.classloader is set to true, and the MR client includes a jetty jar in
its libjars or job jar, the MR app master fails to start. A typical stack trace we get is
as follows:
> {noformat}
> java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration cannot be
cast to org.mortbay.jetty.webapp.Configuration
> 	at org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
> 	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
> 	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> 	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> 	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> 	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> 	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> 	at org.mortbay.jetty.Server.doStart(Server.java:224)
> 	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> 	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
> 	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
> 	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
> 	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)
> {noformat}
> This happens because as part of the MR app master start the jetty classes are loaded
normally through the app classloader, but WebAppContext tries to load the specific Configuration
class via the thread context classloader (which had been set to the user job classloader).

This message was sent by Atlassian JIRA

View raw message