hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5146) application classloader may be used too early to load classes
Date Fri, 12 Apr 2013 18:08:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630421#comment-13630421

Sangjin Lee commented on MAPREDUCE-5146:

It's not entirely clear to me how to add unit tests for this, as it deals with YarnChild.main()...
I didn't find existing unit tests for YarnChild either.

I tested the change with a set of unit tests that exercise mapreduce (with mapreduce.job.classloader
enabled), and can confirm that it no longer tries to load those com.sun.org.apache.xerces.*
classes outside the job execution.
> application classloader may be used too early to load classes
> -------------------------------------------------------------
>                 Key: MAPREDUCE-5146
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5146
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>    Affects Versions: 2.0.3-alpha
>            Reporter: Sangjin Lee
>            Priority: Minor
>         Attachments: MAPREDUCE-5146.patch
> At least in the case of YarnChild, the application classloader is set fairly early (both
in Configuration and as a TCCL). This has an effect of using the application classloader unexpectedly
> There is a fair amount of code that gets invoked between setting the classloader and
executing mapper/reducer task.
> For example, I saw that the application classloader was asked to load a DOM parser class
(com.sun.org.apache.xerces...) as part of initializing the filesystem. Luckily, in most cases
this would be delegated to the parent classloader as the job classpath would not have those
> However, in general, this behavior carries the risk of loading classes with the app classloader
accidentally, and potentially causing problems such as ClassCastException. Those would turn
into nasty bugs that are hard to fix.
> It would be good to either set the application classloader as late as possible or place
clearer limitations so it loads only the mapper/reducer classes and their dependencies.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message