hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Rosenstrauch (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1938) Ability for having user's classes take precedence over the system classes for tasks' classpath
Date Fri, 17 Dec 2010 18:42:06 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972587#action_12972587

David Rosenstrauch commented on MAPREDUCE-1938:

As per Mahadev's comment ("I have usually seens users wanting to override one of the jars
in the classpath.")  That's my use case and, I'd think, the major use case re: this bug.

To clarify my issue:  the version of Jackson that ships with Hadoop is old.  But Avro, which
I use in several of my M/R jobs, requires a newer version.  Even if I supply the newer Jackson
jar using -libjars, the job still picks up the old one (which has priority on the classpath)
and so my job fails.

There is no easy workaround for this.  What we wound up doing to fix this issue in our cluster
was to go into every single node, rename the jackson.jar out of the way, and then restart
the daemons.  Very manual and error prone process.  (Not to mention a very inelegant fix.)

So what's needed here is some way to tell Hadoop (without too much muss and fuss):  for execution
of this M/R job only, *my* version of the jackson.jar takes precedence over the one that comes

> Ability for having user's classes take precedence over the system classes for tasks'
> ----------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1938
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: job submission, task, tasktracker
>            Reporter: Devaraj Das
>            Assignee: Krishna Ramachandran
>            Priority: Blocker
>             Fix For: 0.22.0
>         Attachments: mapred-1938-1, mapred-1938-2.patch, mapred-1938-3.patch, mr-1938-bp20.1.patch,
> It would be nice to have the ability in MapReduce to allow users to specify for their
jobs alternate implementations of classes that are already defined in the MapReduce libraries.
For example, an alternate implementation for CombineFileInputFormat. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message