From user-return-2629-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 4 00:57:31 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 01BF9DD10 for ; Sun, 4 Nov 2012 00:57:31 +0000 (UTC) Received: (qmail 59430 invoked by uid 500); 4 Nov 2012 00:57:26 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 59317 invoked by uid 500); 4 Nov 2012 00:57:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 59309 invoked by uid 99); 4 Nov 2012 00:57:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Nov 2012 00:57:26 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jayunit100@gmail.com designates 209.85.220.176 as permitted sender) Received: from [209.85.220.176] (HELO mail-vc0-f176.google.com) (209.85.220.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Nov 2012 00:57:18 +0000 Received: by mail-vc0-f176.google.com with SMTP id fo13so579481vcb.35 for ; Sat, 03 Nov 2012 17:56:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=pRczAO0K7JnttAbkUFSn/0iTb/7DSVXTYdW6J/K1agw=; b=aObnsYGLVRLNByqL1M5GK//hOKcyrdyNdaajOd942/Tgj6L+/p5kthOeelFvRCkZS+ Z+drjOSrUoG67zsb7A4owv1ywx1xz/0yPdNoE7ibMBrcuUEMG0bYipY/4wgvxmUzRYNs TZTr4KqLd1WfWlQmalnKtp+8RCH/EYqCCS9kBdS6TH/LQlni6XVLfjwak1QtYHfm5yv9 U/ctXDqG1WBsw3Uy7iiXtcu+7xDeb4fmG687mZ1kl/eyyWr83zuFU+cD7sKXzWr/BY7W 9PuskkucY3KBdIbHRFed2tPflRXd4/UHUMjxmTNl6v8oh+YojLrlRbv7rIS3Jvgds6nG eoLg== Received: by 10.52.65.147 with SMTP id x19mr4997968vds.113.1351990617756; Sat, 03 Nov 2012 17:56:57 -0700 (PDT) Received: from [10.67.160.76] (mobile-198-228-195-237.mycingular.net. [198.228.195.237]) by mx.google.com with ESMTPS id g5sm6845501vez.6.2012.11.03.17.56.55 (version=SSLv3 cipher=OTHER); Sat, 03 Nov 2012 17:56:57 -0700 (PDT) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: multipart/alternative; boundary=Apple-Mail-0A3C4E7D-A9F5-449D-9323-8905A1AE41A7 Content-Transfer-Encoding: 7bit Message-Id: Cc: "user@hadoop.apache.org" X-Mailer: iPhone Mail (10A523) From: JAX Subject: Re: hadoop jar question Date: Sat, 3 Nov 2012 20:56:55 -0400 To: "user@hadoop.apache.org" X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-0A3C4E7D-A9F5-449D-9323-8905A1AE41A7 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Ahh yes - i once found that the cause of this was an old version of hadoop w= hich was created using an obsolete but still functional emr script creator..= but anyways- If you want to manually override jars in jobs you can- You ha= ve to override the class path using the specific classpath option ... See th= is post http://stackoverflow.com/questions/11685949/overriding-default-hadoop-jars-= in-class-path... Jay Vyas=20 MMSB UCHC On Nov 3, 2012, at 7:15 PM, Koert Kuipers wrote: > i am looking at the code for RunJar.java which is behind "hadoop jar" for h= adoop 0.20.2 (from cdh3u5). >=20 > i see > 1) jar is unpacked to a temporary directory > 2) the file URLs of all the jars found in the lib subdir of the unpacked j= ar are gathered into a list called classPath > 3) a new ClassLoader loader is created of type URLClassLoader using the li= st of URLs classPath from 2) > 4) the classloader for the current thread is set to newly created loader > 5) the main method of the user provided class is launched using the new lo= ader >=20 > given all this, i would say the new loader will be consulted first (locall= y, not on cluster) when my classes need to be loaded, which means classes fo= und in the jars inside the lib folder of the provided jar should get prefere= nce. yet what i observe is that the classes from ancient jackson jar that is= included with hadoop get preference over the ones i includ (by putting newe= r jackson jars inside the lib folder of my jar). why is this? what am i miss= ing? >=20 > thanks! --Apple-Mail-0A3C4E7D-A9F5-449D-9323-8905A1AE41A7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Ahh yes - i once found that the cause o= f this was an old version of hadoop which was created using an obsolete but s= till functional emr script creator..

but anyways- I= f you want to manually override jars in jobs  you can-  You have t= o override the class path using the specific classpath option ... See this p= ost

i= am looking at the code for RunJar.java which is behind "hadoop jar" for had= oop 0.20.2 (from cdh3u5).

i see
1) jar is unpacked to a  temp= orary directory
2) the file URLs of all the jars found in the lib subdir o= f the unpacked jar are gathered into a list called classPath
3) a new ClassLoader loader is created of type URLClassLoader using the list= of URLs classPath from 2)
4) the classloader for the current thread is s= et to newly created loader
5) the main method of the user provided class i= s launched using the new loader

given all this, i would say the new loader will be consulted first (loca= lly, not on cluster) when my classes need to be loaded, which means classes f= ound in the jars inside the lib folder of the provided jar should get prefer= ence. yet what i observe is that the classes from ancient jackson jar that i= s included with hadoop get preference over the ones i includ (by putting new= er jackson jars inside the lib folder of my jar). why is this? what am i mis= sing?

thanks!
= --Apple-Mail-0A3C4E7D-A9F5-449D-9323-8905A1AE41A7--