hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Segel, Mike" <mse...@navteq.com>
Subject Custom Class Loader for Hadoop M/R jobs?
Date Wed, 14 Apr 2010 13:39:50 GMT

Ok, here's a bit of a bizarre  issue...

How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party

An example: Hadoop has an older version of an open source jar in its /lib directory. You're
interfacing with a 3rd party OS tool that uses a later release of the same jar.

You can modify the classpath, and that might work. But the better way is to create a Custom
Class Loader. (Non-trivial)

Looking at the Configuration class, it looks like there are a couple of methods that deal
with loading a class in to the configuration so that the m/r jobs can have access to them
on each node.

Is this the correct intended use, or am I missing something?
Has anyone done something like this?



Michael Segel
Architect,  R&D
425 West Randolph Street
Chicago, IL 60606
(T)  +1 312-780-3432
(C)  +1 312-952-8175

The information contained in this communication may be CONFIDENTIAL and is intended only for
the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this communication in error, please
notify the sender and delete/destroy the original message and any copy of it from your computer
or paper files.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message