Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 67068D2D1 for ; Wed, 3 Oct 2012 12:02:16 +0000 (UTC) Received: (qmail 78540 invoked by uid 500); 3 Oct 2012 12:02:10 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 77773 invoked by uid 500); 3 Oct 2012 12:02:05 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77743 invoked by uid 99); 3 Oct 2012 12:02:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Oct 2012 12:02:04 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jayunit100@gmail.com designates 209.85.220.48 as permitted sender) Received: from [209.85.220.48] (HELO mail-pa0-f48.google.com) (209.85.220.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Oct 2012 12:01:59 +0000 Received: by pabkp12 with SMTP id kp12so8101147pab.35 for ; Wed, 03 Oct 2012 05:01:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hhZASxeJ/JFi9X/OvRxu0yW2k+uVuzLoOZyUu8BE/lI=; b=S/B82U4DRysGG3OgHD71sIHoCiVwn3SR5MdR1UwA0rMum37IrFdZid4bWLTLwbPIoN 7z7okA/oElVAFf39rc5yPrqbIWMHtxdFV0lMrN252TrMdQuuaH7RinZlT76/c97+n6fC fdmwe5fbmfSsc+wV2YJmILE9IVdQ745/MZSQtoLhLO15cU5xn4pJ6AZDmPFqVTHikEjV HkPWttbN6dXQ1QxskAlD9Q8+BlCdi+kjN02fjRcfnjJ5YDPrCyFB31be8Kp1pyD199qa MF1ehvqolZfIyXXrpfvlEYx6ZWEwlBMNbJ5VR5FS/HwwoJkk8gvFYJW1iFBDjoRtbUKx 28Ug== MIME-Version: 1.0 Received: by 10.68.132.41 with SMTP id or9mr12537140pbb.67.1349265699135; Wed, 03 Oct 2012 05:01:39 -0700 (PDT) Received: by 10.68.20.194 with HTTP; Wed, 3 Oct 2012 05:01:39 -0700 (PDT) In-Reply-To: References: Date: Wed, 3 Oct 2012 08:01:39 -0400 Message-ID: Subject: Re: Lib conflicts From: Jay Vyas To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b10c86db4610904cb2666e8 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b10c86db4610904cb2666e8 Content-Type: text/plain; charset=ISO-8859-1 Yup! I hate this issue. It also happens with json libs if you have an old hadoop !!! Its really easy also to dump the exact class path at runtime using the ((URLClassLoader)ClassLoader.getSystemClassLoader()).getURLs(); trick... very very very very useful :) I always do this just to make sure before going and mucking with the classpath args. I've found using this trick can save a lot of headaches, especially if your dealing with jars that you can't programmatically get version #'s out of. On Wed, Oct 3, 2012 at 7:45 AM, Ben Rycroft wrote: > > Hi guys, > > Thanks the speedy replies. > > I will try Harsh's suggestion and see if this solves the issue, if not I will just do what Michael suggested and replace the jars on each of the nodes. > > Thanks again! > > > On Wed, Oct 3, 2012 at 12:39 PM, Harsh J wrote: >> >> Hi Ben, >> >> As long as the switch of libraries doesn't impact the execution of the >> Child task code itself, for Apache Hadoop 1.x, using the config >> "mapreduce.user.classpath.first" set to true may solve your trouble. >> >> On Wed, Oct 3, 2012 at 4:51 PM, Ben Rycroft wrote: >> > Hi all, >> > >> > I have a jar that uses the Hadoop API to launch various remote mapreduce >> > jobs (ie, im not using the command-line to initiate the job). The service >> > jar that executes the various jobs is built with maven's >> > "jar-with-dependencies". >> > >> > My jobs all run fine except one that uses commons-codec 1.7, I get: >> > >> > FATAL org.apache.hadoop.mapred.Child: Error running child : >> > java.lang.NoSuchMethodError: >> > org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String; >> > >> > I think this is because my jar is including commons-codec 1.7 whereas my >> > Hadoop install's lib has commons-codec 1.4 ... >> > >> > Is their any way to instruct Hadoop to use the distributed commons-codec 1.7 >> > (I assume this is distributed as a job dependency) rather than the >> > commons-codec 1.4 in the hadoop 1.0.3 core lib? >> > >> > Removing commons-codec-1.4.jar from my Hadoop library folder did seem to >> > solve the problem for a bit, but is not working on another VM. Replacing the >> > 1.4 jar with the 1.7 does seem to fix the problem but this doesn't seem too >> > sane. Hopefully there is a better alternative. >> > >> > Thanks! >> >> >> >> -- >> Harsh J > > -- Jay Vyas MMSB/UCHC --047d7b10c86db4610904cb2666e8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Yup! I hate this issue.=A0 It also happens with json libs if you have an ol= d hadoop !!!

Its really easy also to dump the exact class path at r= untime using the

((URLClassLoader)ClassLoader.getSystemClassLoader()= ).getURLs(); trick... very very very very useful :)

I always do this just to make sure before going and mucking with the cl= asspath args.

I've found using this trick can save a lot of head= aches, especially if your dealing with jars that you can't programmatic= ally get version #'s out of.

On Wed, Oct 3, 2012 at 7:45 AM, Ben Rycroft <brycroft89@gmail.com> wrote:
>
> Hi guy= s,
>
> Thanks the speedy replies.
>
> I will try Ha= rsh's suggestion and see if this solves the issue, if not I will just d= o what Michael suggested and replace the jars on each of the nodes.
>
> Thanks again!
>
>
> On Wed, Oct 3, 2012 at 1= 2:39 PM, Harsh J <harsh@cloudera.c= om> wrote:
>>
>> Hi Ben,
>>
>> A= s long as the switch of libraries doesn't impact the execution of the >> Child task code itself, for Apache Hadoop 1.x, using the config>> "mapreduce.user.classpath.first" set to true may solve = your trouble.
>>
>> On Wed, Oct 3, 2012 at 4:51 PM, Ben R= ycroft <brycroft89@gmail.com= > wrote:
>> > Hi all,
>> >
>> > I have a jar that u= ses the Hadoop API to launch various remote mapreduce
>> > jobs= (ie, im not using the command-line to initiate the job). The service
>> > jar that executes the various jobs is built with maven's<= br>>> > "jar-with-dependencies".
>> >
>= ;> > My jobs all run fine except one that uses commons-codec 1.7, I g= et:
>> >
>> > FATAL org.apache.hadoop.mapred.Child: Error = running child :
>> > java.lang.NoSuchMethodError:
>> &= gt; org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/Str= ing;
>> >
>> > I think this is because my jar is including = commons-codec 1.7 whereas my
>> > Hadoop install's lib has = commons-codec 1.4 ...
>> >
>> > Is their any way to= instruct Hadoop to use the distributed commons-codec 1.7
>> > (I assume this is distributed as a job dependency) rather tha= n the
>> > commons-codec 1.4 in the hadoop 1.0.3 core lib?
&= gt;> >
>> > Removing commons-codec-1.4.jar from my Hadoop= library folder did seem to
>> > solve the problem for a bit, but is not working on another VM= . Replacing the
>> > 1.4 jar with the 1.7 does seem to fix the = problem but this doesn't seem too
>> > sane. Hopefully ther= e is a better alternative.
>> >
>> > Thanks!
>>
>>
>><= br>>> --
>> Harsh J
>
>



--
Jay= Vyas
MMSB/UCHC --047d7b10c86db4610904cb2666e8--