hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
Date Wed, 02 Jul 2014 05:14:24 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049632#comment-14049632
] 

Sangjin Lee commented on MAPREDUCE-5957:
----------------------------------------

This is a regression introduced by MAPREDUCE-5751.

MAPREDUCE-5751 moved setting of the job classloader from before the MRAppMaster init to the
end of the MRAppMaster start. However, some classes are compelled from the configuration in
between these two points, such as the output format class, the speculator class, etc. The
job classloader must be used to load those classes or they will not be found.

> AM throws ClassNotFoundException with job classloader enabled if custom output format/committer
is used
> -------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>
> With the job classloader enabled, the MR AM throws ClassNotFoundException if a custom
output format class is specified.
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
> 	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat
not found
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
> 	at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
> 	... 8 more
> Caused by: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not
found
> 	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
> 	... 10 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message