hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henning Blohm (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
Date Thu, 04 Nov 2010 13:35:48 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928208#action_12928208

Henning Blohm commented on MAPREDUCE-1700:

The patch in MAPREDUCE-1938 does unfortunately not solve the issue when the job implementation
uses custom class loaders to load dependency classes. The proposed patch only addresses the
issue when no custom class loaders are in the picture.

As a first step, it would really help (us anyway), if jobs would not be started with stuff
on the classpath that is not at all required for job execution per se (e.g. jetty libs, the
eclipse java compiler, jasper...).

Secondly, hadoop could actually start with only the hadoop api types on the classpath plus
a small launcher that would load hadoops implementation in an isolated child loader, so that
implementation dependencies do not leak through to the job's implementation. I am not sure
if the hadoop implementation is ready for implementation/api separation via class loaders

I patched hadoop 0.20.2 to exclude all libs in lib/server from the jobs classpath and I move
all non-job related jars into that server folder in my hadoop installation. That helped somewhat.

> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
> If user code has a dependency on a version of a JAR that is different to the one that
happens to be used by Hadoop, then it may not work correctly. This happened with user code
using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using
a specialized classloader in the Child JVM is probably the way to solve this.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message