hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom White <...@cloudera.com>
Subject Re: Custom Class Loader for Hadoop M/R jobs?
Date Wed, 14 Apr 2010 14:56:17 GMT
I agree that it writing a custom classloader is non-trivial (and I
don't know if anyone has done this in the MapReduce context), so this
would best be handled by the framework. I've opened
https://issues.apache.org/jira/browse/MAPREDUCE-1700 for this.

Tom

On Wed, Apr 14, 2010 at 6:39 AM, Segel, Mike <msegel@navteq.com> wrote:
> Hi,
>
> Ok, here's a bit of a bizarre  issue...
>
> How do you handle class collisions between Hadoop and your m/r job which calls other
3rd party classes.
>
> An example: Hadoop has an older version of an open source jar in its /lib directory.
You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>
> You can modify the classpath, and that might work. But the better way is to create a
Custom Class Loader. (Non-trivial)
>
> Looking at the Configuration class, it looks like there are a couple of methods that
deal with loading a class in to the configuration so that the m/r jobs can have access to
them on each node.
>
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>
> Thx
>
> -Mike
>
> Michael Segel
> Architect,  R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T)  +1 312-780-3432
> (C)  +1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only
for the use of the recipient(s) named above.  If you are not the intended recipient, you
are hereby notified that any dissemination, distribution, or copying of this communication,
or any of its contents, is strictly prohibited.  If you have received this communication
in error, please notify the sender and delete/destroy the original message and any copy of
it from your computer or paper files.
>

Mime
View raw message