hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
Date Wed, 16 Jul 2014 20:35:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064044#comment-14064044

Jason Lowe commented on MAPREDUCE-5957:

Thanks for updating the patch.  I do think the wrapper makes it a bit clearer we're calling
something within the context of the job classloader.

One of the drawbacks of the wrapper is that it's harder to deal with checked exceptions. 
For example, MRAppMaster.isRecoverySupported used to propagate any IOException thrown by the
user-provided committer code, but the wrapper now transforms this into a runtime exception.
 Not sure it really makes a whole lot of difference in practice for that case, but it is a
subtle change in behavior.  Do you think it's worth adding a wrapper form that can throw Exception
(or maybe just IOException)?  Thinking along similar lines as PrivilegedAction vs. PrivilegedExceptionAction
in the doAs scenarios.  I guess callers can catch YarnRuntimeException with the current patch
and check the cause if they want/need to convert it back to a checked exception themselves,
but curious on your thoughts.

Otherwise I think the patch looks good.  Couple of minor additional items:

- Is there a reason the init transition calls MRAppMaster.createJobClassloader rather than
having MRAppMaster.serviceInit do it?  Seems more natural to have the serviceInit method initialize
the MRAppMaster data members if possible.
- Not strictly related to this JIRA, but I happened to notice that MRAppMaster.isRecoverySupported
takes a committer argument that it totally ignores.  Since we're in the area, it would be
nice to clean that up by either removing the argument or use it rather than the committer
member.  Totally OK with me if you'd rather postpone that minor cleanup to another JIRA.

> AM throws ClassNotFoundException with job classloader enabled if custom output format/committer
is used
> -------------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
> With the job classloader enabled, the MR AM throws ClassNotFoundException if a custom
output format class is specified.
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
> 	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat
not found
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
> 	at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
> 	... 8 more
> Caused by: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not
> 	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
> 	... 10 more
> {noformat}

This message was sent by Atlassian JIRA

View raw message