hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1938) Ability for having user's classes take precedence over the system classes for tasks' classpath
Date Wed, 14 Jul 2010 16:57:50 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888433#action_12888433

Doug Cutting commented on MAPREDUCE-1938:

Two thoughts:
 1. In general, we need to better separate the kernel from the library.  CombineFileInputFormat
is library code and should be easy to update without updating the cluster.  Long-term, only
kernel code should be hardwired on the classpath of tasks, with library and user code both
specified per job.  There should be no default version of library classes for a task: tasks
should always specify their required libraries.  Is there a Jira for this?  I know Tom's expressed
interest in working on this.
 2. We should permit user code to depend on different versions of things than the kernel does.
 For example, user code might rely on a different version of HttpClient or Avro than that
used by MapReduce.  This should be possible if instances of classes from these are not a passed
between user and kernel code, e.g., as long as Avro and HttpClient classes are not a part
of the MapReduce API.  In this case classloaders (probably via OSGI) could permit this.

> Ability for having user's classes take precedence over the system classes for tasks'
> ----------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1938
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: job submission, task, tasktracker
>            Reporter: Devaraj Das
>             Fix For: 0.22.0
>         Attachments: mr-1938-bp20.patch
> It would be nice to have the ability in MapReduce to allow users to specify for their
jobs alternate implementations of classes that are already defined in the MapReduce libraries.
For example, an alternate implementation for CombineFileInputFormat. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message