hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
Date Thu, 06 Sep 2012 15:47:09 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated MAPREDUCE-1700:

    Attachment: MAPREDUCE-1700-ccl.patch

Prompted by this discussion I had a look at using a classloader approach similar to how servlet
containers are implemented. The servlet spec says that classes in the WEB-INF/classes directory
and JARs in the WEB-INF/lib directory are loaded in preference to system classes. I found
this page about classloading in Jetty useful: http://docs.codehaus.org/display/JETTY/Classloading.

The attached patch does a similar thing for the Hadoop task classpath by using a custom classloader
for classes instantiated by reflection in MapTask. The unit test from the previous patch passes
with this implementation. I think this is worth exploring further.

> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch
> If user code has a dependency on a version of a JAR that is different to the one that
happens to be used by Hadoop, then it may not work correctly. This happened with user code
using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using
a specialized classloader in the Child JVM is probably the way to solve this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message