hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1478) Separate the mapred.lib and mapreduce.lib classes to a different jar and include the user jar ahead of the lib jar.
Date Tue, 18 Jan 2011 19:04:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983334#action_12983334

Scott Carey commented on MAPREDUCE-1478:

This approach will work.

I see another possibility that is simpler, but has some serious drawbacks.

Instead of separating the source directories and classpaths at build time, file set includes/excludes
could be used at jar creation time to create two jar files.
Benefit:  simpler, requires no code changes:
Drawback:  does not actually disentangle the code, which is more brittle and could lead to
confusion depending on what changes are in the user lib jars.

That is a big drawback.  

The above approach is more complicated but does enforce true separation.
In the long run, tackling the disentangling of the lib from the system is a requirement but
the extra complexity requires more careful review.  It also is the bulk of the work required
to make the lib a truly separate library -- for maven or OSGi.

I'm not an expert on the details of the API and code changes needed for clean separation as
a consequence of this.  The callback changes seem like an improvement.  Duplication of wrappers
and reflective access are usually signs that there is a better way, but that better way almost
always requires an API break.

Overall, I like this approach because it is moving in the right direction for separating the
core execution from more volatile or user-extensible library code. 

> Separate the mapred.lib and mapreduce.lib classes to a different jar and include the
user jar ahead of the lib jar.
> -------------------------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1478
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1478
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>            Reporter: Owen O'Malley
>         Attachments: MAPREDUCE-1478.patch, move.sh
> Currently the user can't include updated library jars as part of their job. By pulling
out the lib classes we can include the classes (eg. TextInputFormat) in the user's jar and
get their version and not the system installed one.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message