hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
Date Thu, 04 Oct 2012 07:33:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469203#comment-13469203

Alejandro Abdelnur commented on MAPREDUCE-1700:


What I would add is the capability of blacklisting packages. This is, if a package is blacklisted
and a class under that package hierarchy is found in the job JARs, the job should fail. This
is something avail in webapp classloaders to avoid webapps for bundling things like servlet/jsp
JARs that would break things. In our case we would blacklist common/hdfs/yarn/mapred packages
and log4j (the factory is a singleton and if present in the job JARs will trash the log configuration
of hadoop). I could see other JARs fitting this blacklist, thus I'd suggest that we have a
config property with the list of blacklisted packages.

This is isolating MR jobs from Hadoop JARs. I think we should do the same at YARN level to
isolate YARN JARs from AM JARs. Because of this, the JobClassLoader should be in common and
probably have a different name, like IsolationClassLoader. Also it should receive, in the
constructor, the blacklist.

> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch,
> If user code has a dependency on a version of a JAR that is different to the one that
happens to be used by Hadoop, then it may not work correctly. This happened with user code
using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using
a specialized classloader in the Child JVM is probably the way to solve this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message