hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
Date Thu, 06 Sep 2012 05:22:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449430#comment-13449430

Luke Lu commented on MAPREDUCE-1700:

bq. Without a user doing classloader gymnasitics and fancy packaging themselves, there is
not always a way.

That's an interesting way to say that except for some ways that would always work, there is
not always a way. Using the standard task API to bootstrap an OSGi container is reasonably
straight forward :)

bq. A user cannot simply package a jar up and ask hadoop to execute it and expose to the user's
execution environment only the public Hadoop API.

I do agree that there is a usability issue for certain (and arguably less common) use cases,
where a user wants to use dependencies that conflict with client framework. However the proposed
OSGi approach makes the usability worse for common cases: You'll always need OSGi bundles,
which is a form of "fancy packaging", to run your jobs.

A more reasonable (and less heavy) solution would not require users to make any change (including
adding metadata to their jars) to their existing code.
> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-1700.patch, MAPREDUCE-1700.patch
> If user code has a dependency on a version of a JAR that is different to the one that
happens to be used by Hadoop, then it may not work correctly. This happened with user code
using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using
a specialized classloader in the Child JVM is probably the way to solve this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message