hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5146) application classloader may be used too early to load classes
Date Mon, 22 Apr 2013 20:07:18 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated MAPREDUCE-5146:

    Attachment: classloader-test.patch

+1 for the patch.

I've attached a test that I used when writing MAPREDUCE-1700. I couldn't find a simple way
of integrating it in the Hadoop build, which is why it wasn't checked in there. I just ran
it now with the patch for this issue and it passed.
> application classloader may be used too early to load classes
> -------------------------------------------------------------
>                 Key: MAPREDUCE-5146
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5146
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>    Affects Versions: 2.0.3-alpha
>            Reporter: Sangjin Lee
>            Priority: Minor
>             Fix For: 2.0.5-beta
>         Attachments: classloader-test.patch, MAPREDUCE-5146.patch
> At least in the case of YarnChild, the application classloader is set fairly early (both
in Configuration and as a TCCL). This has an effect of using the application classloader unexpectedly
> There is a fair amount of code that gets invoked between setting the classloader and
executing mapper/reducer task.
> For example, I saw that the application classloader was asked to load a DOM parser class
(com.sun.org.apache.xerces...) as part of initializing the filesystem. Luckily, in most cases
this would be delegated to the parent classloader as the job classpath would not have those
> However, in general, this behavior carries the risk of loading classes with the app classloader
accidentally, and potentially causing problems such as ClassCastException. Those would turn
into nasty bugs that are hard to fix.
> It would be good to either set the application classloader as late as possible or place
clearer limitations so it loads only the mapper/reducer classes and their dependencies.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message