Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D90A410285 for ; Tue, 27 Aug 2013 21:32:17 +0000 (UTC) Received: (qmail 536 invoked by uid 500); 27 Aug 2013 21:32:12 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 428 invoked by uid 500); 27 Aug 2013 21:32:12 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 421 invoked by uid 99); 27 Aug 2013 21:32:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Aug 2013 21:32:12 +0000 X-ASF-Spam-Status: No, hits=2.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of java8964@hotmail.com designates 65.55.111.82 as permitted sender) Received: from [65.55.111.82] (HELO blu0-omc2-s7.blu0.hotmail.com) (65.55.111.82) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Aug 2013 21:32:06 +0000 Received: from BLU162-W58 ([65.55.111.72]) by blu0-omc2-s7.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 27 Aug 2013 14:31:45 -0700 X-TMN: [SLVtH1nLbE1amd1H43ruPqDTHQ+bTS3l9Ti+oPRwFdo=] X-Originating-Email: [java8964@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_58d43d60-8677-460e-b367-c9d0da4a2f20_" From: java8964 java8964 To: "user@hadoop.apache.org" Subject: RE: Jar issue Date: Tue, 27 Aug 2013 17:31:45 -0400 Importance: Normal In-Reply-To: References: ,, MIME-Version: 1.0 X-OriginalArrivalTime: 27 Aug 2013 21:31:45.0866 (UTC) FILETIME=[D4951AA0:01CEA36C] X-Virus-Checked: Checked by ClamAV on apache.org --_58d43d60-8677-460e-b367-c9d0da4a2f20_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I am not sure the original suggestion will work for your case. My understanding is the you want to use some API=2C only exists in slf4j ve= rsiobn 1.6.4=2C but this library with different version already existed in = your hadoop environment=2C which is quite possible. To change the maven build of the application maybe not work due to: 1) If some API of 1.6.4 being used in the application code=2C then it must = be shipped with your code into hadoop cluster.2) What you are looking for m= aybe is this parameter "mapreducer.user.classpath.first"=2C you can google = it=2C which allow the user's class files loaded before the Hadoop's in mapp= er/reducer tasks.3) Keep in mind that even your code should be fine now=2C = but if the library jar you submit MAYBE not backward compatibility=2C then= it may cause problem in the Mapper/Reducer tasks for Hadoop code=2C as now= the slf4j 1.6.4 version of class code being loaded into JVM. But this case= is very rare=2C as a lot of you time you will submit a later version of ja= r than what being contained in hadoop=2C and most of them are backward comp= atibility.4) Maybe Hadoop can use different classloader to load the user's = jar file=2C like OLD J2EE container did=2C to solve this kind of problem in= the future.Yong Date: Tue=2C 27 Aug 2013 09:05:51 -0700 Subject: Re: Jar issue From: jamalshasha@gmail.com To: user@hadoop.apache.org I am right now using libjars option.How do i do what you suggested using th= at route? On Tue=2C Aug 27=2C 2013 at 8:51 AM=2C Shahab Yunus wrote: =0A= One idea is=2C you can use the exclusion property of maven (provided you ar= e using that to build your application) while including hadoop dependencies= and exclude sl4j that is coming within hadoop and then include your own sl= 4j as a separate dependency. Something like this:=0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= org.apache.hbase =0A= =0A= hbase =0A= =0A= 0.92.1-cdh4.1.2 =0A= =0A= =0A= =0A= log4j =0A= =0A= log4j =0A= =0A= =0A= =0A= slf4j-log4j12 =0A= =0A= org.slf4j =0A= =0A= =0A= =0A= Regards=2CShahab=0A= = --_58d43d60-8677-460e-b367-c9d0da4a2f20_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I am not sure the original suggestion will work for your case.

My understanding is the you want to use some API=2C= only exists in slf4j versiobn 1.6.4=2C but this library with different ver= sion already existed in your hadoop environment=2C which is quite possible.=

To change the maven build of the application maybe= not work due to:

=
1) If some API of 1.6.4 being use= d in the application code=2C then it must be shipped with your code into ha= doop cluster.
2) What you ar= e looking for maybe is this parameter "mapreducer.user.classpath.first"=2C = you can google it=2C which allow the user's class files loaded before the H= adoop's in mapper/reducer tasks.
3) Keep in mind that= even your code should be fine now=2C but if the library  =3Bjar you su= bmit MAYBE not backward compatibility=2C then it may cause problem in the M= apper/Reducer tasks for Hadoop code=2C as now the slf4j 1.6.4 version of cl= ass code being loaded into JVM. But this case is very rare=2C as a lot of y= ou time you will submit a later version of jar than what being contained in= hadoop=2C and most of them are backward =3Bcompatibility.
4) Maybe Hadoop ca= n use different classloader to load the user's jar file=2C like OLD J2EE co= ntainer did=2C to solve this kind of problem in the future.

Yong

Dat= e: Tue=2C 27 Aug 2013 09:05:51 -0700
Subject: Re: Jar issue
From: jam= alshasha@gmail.com
To: user@hadoop.apache.org

I = am right now using libjars option.
How do i do what you suggested using= that route?


On Tue=2C Aug 27=2C 2013 at 8:51 AM=2C Shah= ab Yunus <=3Bshahab.yunus@gmail.com>=3B wrote:
=0A=
One idea is=2C you can use the exclus= ion property of maven (provided you are using that to build your applicatio= n) while including hadoop dependencies and exclude sl4j that is coming with= in hadoop and then include your own sl4j as a separate dependency. Somethin= g like this:
=0A= =0A=
=0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= <=3Bdependency>=3B
=0A= <=3BgroupId<= /span>>=3Borg.apache.hbase<=3B/groupId<= /span>>=3B
=0A= =0A= <=3Ba= rtifactId>=3Bhbase<=3B/artifactI= d>=3B
=0A= =0A= <=3Bversion<= /span>>=3B0.92.1-cdh4.1.2<=3B/version>=3B
=0A= =0A= <=3Be= xclusions>=3B
=0A= = <=3Bexclusion>=3B
=0A= <=3BartifactId>=3Blog4j<= span><=3B/artifactId>=3B
=0A= =0A= <=3BgroupId>=3Blog4j<=3B/groupId>=3B
=0A= =0A= = <=3B/exclusion>=3B
=0A= = <=3Bexclusion>=3B
=0A= <=3BartifactId>=3Bslf4j-log4j12= <=3B/artifactId>=3B
=0A= =0A= <=3BgroupId>=3Borg.slf4j<=3B/= groupId>=3B
=0A= =0A= = <=3B/exclusion>=3B
=0A= <=3B/exclusions>=3B

=0A= <=3B/dependency>=3B

Regards=2C
Shahab
=0A=

= = --_58d43d60-8677-460e-b367-c9d0da4a2f20_--