Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Thu, 6 Sep 2012 03:57:09 +1100 (NCT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <816763568.39641.1346864229673.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may
 conflict with MapReduce system JARs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448882#comment-13448882 ] 

Luke Lu commented on MAPREDUCE-1700:
------------------------------------

bq. The conflict might require a user jar that is not compatible with one needed by the framework, either order breaks something

You can always change the client framework and make it work with user code, per job, with class path ordering. There is currently always a way in both Hadoop 1 and 2 to submit a job with arbitrary dependencies, even though it might not be pretty (may require change to client framework).

bq. The user might override a system jar and alter functionality in a way that breaks the framework, or subverts security.

The client framework code can always be changed per job to accommodate new dependencies. MR security is done at protocol level, i.e. no amount class path ordering can subvert security.

I agree with Arun that this is a nice to have feature to improve usability. Advanced users can already achieve whatever that can be achieved (including running an OSGi container) per job.
 
                
> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-1700.patch, MAPREDUCE-1700.patch
>
>
> If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira