Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of invisiblesun@mac.com
 designates 17.148.16.105 as permitted sender)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
From: Julia Smith <invisiblesun@mac.com>
In-reply-to: 
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
Date: Wed, 14 Apr 2010 10:55:11 -0700
Message-id: <19CC33EF-FA84-46CB-B61D-385CB92688A5@mac.com>
References: 
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
To: general@hadoop.apache.org

This might be of use to you, as I had a related problem years ago:

http://www.developertutorials.com/tutorials/java/simplify-application-delivery-one-jar-050422/page5.html


The trick is a one-jar solution allows you to embed jar files and load classes from them. Rolling up all your jar variants in a single jar and then selectively loading the desired version might provide you a useful solution. The source is there, so you could start with a working classloader and perhaps apply a few tweaks.

On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:

> Scott,
> 
> While that may work for a quick fix. Its not a good long term solution and you then run in to a problem where you upgrade your hadoop release and the removed jar is replaced or if you replace the jar, it possible to get overwritten.
> 
> In this specific instance, the Jackson libraries are not that important and they can be replaced.
> But that doesn't mean that this issue won't come up again and its something you can't easily pop out and replace.
> 
> This is why I'm looking at custom class loading and trying to understand what can be accomplished with the methods in the Configuration class. 
> 
> Thx
> 
> -Mike
> 
> 
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com] 
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> 
> Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
> 
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things.  However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
> 
> -Scott
> 
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> 
>> Hi,
>> 
>> Ok, here's a bit of a bizarre  issue...
>> 
>> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>> 
>> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>> 
>> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>> 
>> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>> 
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>> 
>> Thx
>> 
>> -Mike
>> 
>> Michael Segel
>> Architect,  R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T)  +1 312-780-3432
>> (C)  +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>> 
>> 
>> 
>> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
> 
> 
> 
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.