hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
Date Wed, 05 Sep 2012 09:30:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448595#comment-13448595

Tom White commented on MAPREDUCE-1700:

Scott makes a good case for why some kind of classloader isolation is needed.

The patch is still a work in progress, but the idea is that the OSGi support is optional -
so if you use a regular (non-OSGi) job JAR then it works like it does today, while if your
job JAR is an OSGi bundle (basically a JAR with extra headers in the manifest, and possibly
some embedded dependencies) then it is loaded in an OSGi container in the task JVM. This allows
folks who want to use OGSi to do so while not impacting others. (Hopefully this answers Steve's

>From the point of view of this JIRA, OSGi is simply a means to ensure classloader isolation.
That means that if Jigsaw became a reality, then we could use that instead or as well. OSGi
has many other features, but they are not used for this change. (Note that there are other
ongoing efforts to make Hadoop more OSGi-friendly, covered in HADOOP-7977, and while some
might be helpful for this JIRA (such as HADOOP-6484), none is required.)

Also, in the future OSGi containers could improve container reuse by providing better isolation
between jobs, since bundles can be unloaded, although I haven't spent any time looking at
how that would work in the context of MR.
> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-1700.patch, MAPREDUCE-1700.patch
> If user code has a dependency on a version of a JAR that is different to the one that
happens to be used by Hadoop, then it may not work correctly. This happened with user code
using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using
a specialized classloader in the Child JVM is probably the way to solve this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message