hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1938) Ability for having user's classes take precedence over the system classes for tasks' classpath
Date Thu, 16 Dec 2010 09:38:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972018#action_12972018

Alejandro Abdelnur commented on MAPREDUCE-1938:

IMO this is not the right way of approaching this issue as it will lead to have multiple implementations
of a library in the classpath and that sooner or later breaks things in very odd ways. Following
an example of an issue I've run into few years ago:

 a-1.jar contains classes: A, B (with B using A)
 a-2.jar contains classes: A, C (with C using A)

A classes from a-1.jar and a-2.jar are different.
Kernel 'sometimes' uses class B from a-1.jar
user-app 'sometimes' uses class C from a-2.jar

All this will compile, but will create very odd errors at runtime.

Because of this, I wouldn't provide a mechanism to allow users to do this.

Instead, I think we should implement a task-classloader that only exposes to the application
the MR API and the user is complete control of all the JARs in the app classpath, including
input/output formats, etc, etc. 

I understand this can be a bit tricky because the line of API vs implementation classes is
not that clear as in the servlet API (to bring an example of APIs with implementations doing

> Ability for having user's classes take precedence over the system classes for tasks'
> ----------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1938
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: job submission, task, tasktracker
>            Reporter: Devaraj Das
>            Assignee: Krishna Ramachandran
>             Fix For: 0.22.0
>         Attachments: mapred-1938-1, mapred-1938-2.patch, mapred-1938-3.patch, mr-1938-bp20.1.patch,
> It would be nice to have the ability in MapReduce to allow users to specify for their
jobs alternate implementations of classes that are already defined in the MapReduce libraries.
For example, an alternate implementation for CombineFileInputFormat. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message