hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
Date Wed, 07 Nov 2012 16:51:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated MAPREDUCE-1700:

    Attachment: MAPREDUCE-1700.patch

Thanks for the comments Alejandro.

bq. What I would add is the capability of blacklisting packages. 

I think that is a good idea. Servlet containers do this - e.g. system classes in Jetty are
always loaded from the parent (http://docs.codehaus.org/display/JETTY/Classloading). Rather
than failing the job if the class is a system class and is found in the job classpath (as
you suggested) I think it would be acceptable to log a warning but load from the system classpath.
I expect the default blacklist would be {{java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop.}}.

bq. I think we should do the same at YARN level to isolate YARN JARs from AM JARs. Because
of this, the JobClassLoader should be in common and probably have a different name, like IsolationClassLoader.

Other YARN apps might benefit from this work, so perhaps we should add the classloader to
YARN (not Common, since HDFS shouldn't need it), and the MR-specific parts would stay in MR,
of course.

bq. we should see how to enable the classloader on the client side as well as it may be required
(to use different JARs) for the submission code. May be as another JIRA. 

I think this is a slightly different problem, since users generally have more control over
the JVM they submit from than the JVM the task runs in. So, yes, another JIRA would be appropriate.

bq. Also, don't recall now if it is there or not, we may want o have a job config property
to disable it in case some app runs into funny issues with it.

Agreed. It's off by default in the current patch.
> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch
> If user code has a dependency on a version of a JAR that is different to the one that
happens to be used by Hadoop, then it may not work correctly. This happened with user code
using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using
a specialized classloader in the Child JVM is probably the way to solve this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message