Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9211FD8E7 for ; Thu, 7 Mar 2013 18:48:46 +0000 (UTC) Received: (qmail 26123 invoked by uid 500); 7 Mar 2013 18:48:41 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 25976 invoked by uid 500); 7 Mar 2013 18:48:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 25969 invoked by uid 99); 7 Mar 2013 18:48:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Mar 2013 18:48:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of barak.yaish@gmail.com designates 209.85.128.43 as permitted sender) Received: from [209.85.128.43] (HELO mail-qe0-f43.google.com) (209.85.128.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Mar 2013 18:48:34 +0000 Received: by mail-qe0-f43.google.com with SMTP id 1so488604qee.16 for ; Thu, 07 Mar 2013 10:48:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=MSI/NlOuKc/fwZvz9hTe7DoWxpv9Rqqr1oOsM+tG10U=; b=P+aYnP9B0C7JiInLCiAAjSbRWhYV1Jwa+BvdLFeoFfPw6LByGV5Rr+stYoO11K5jXv F+Pia7Ov5dIzc1H3d5XsxJcN3cVU9c7y6ENxlRcHpjRsTiHlOFDYE2Fez/UTJI4slVWx EbNqYOd4sUa/XIxIaEsjDS5jgXSg+KLeyfJsyYDh/JS6/wrrR2kZWnWp5hgIZDYHdqqr 0AdMeQIUqTJYGB7MBbLQG2jubwffBRiB9HfKYCeET5it6eqMaDIfmIaP7Cl9PQj8bFcp O12JKGTwWtmOQeT7SQ5P52M/elNtW+6nEiofyEukM2qrVZzE7IVr5KnYeQmcErB0pWt5 cNmw== MIME-Version: 1.0 X-Received: by 10.49.94.238 with SMTP id df14mr28213725qeb.44.1362682094231; Thu, 07 Mar 2013 10:48:14 -0800 (PST) Received: by 10.49.18.198 with HTTP; Thu, 7 Mar 2013 10:48:14 -0800 (PST) Date: Thu, 7 Mar 2013 20:48:14 +0200 Message-ID: Subject: Job driver and 3rd party jars From: Barak Yaish To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b67893a2b287904d75a26f3 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b67893a2b287904d75a26f3 Content-Type: text/plain; charset=ISO-8859-1 Hi, I'm able to run M/R jobs where the mapper and reducer required to use 3rd party jars. I'm registering those jars in -libjar while invoking the hadoop jar command. I'm facing a strange problem, though, when the job driver itself ( extends Configured implements Tool ) required to run such code ( for example notify some remote service upon start and end). Is there a way to configure classpath when submitting jobs using hadoop jar? Seems like -libjar doesn't work for this case... Exception in thread "main" java.lang.NoClassDefFoundError: com/me/context/DefaultContext at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) --047d7b67893a2b287904d75a26f3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

I'm able to run M/R jobs where = the mapper and reducer required to use 3rd party jars. I'm registering = those jars in -libjar while invoking the hadoop jar command. I'm facing= a strange problem, though, when the job driver itself (=A0extends Configur= ed implements Tool ) required to run such code ( for example notify some re= mote service upon start and end). Is there a way to configure classpath whe= n submitting jobs using hadoop jar? Seems like -libjar doesn't work for= this case...

Exception in thread "main" java.lang.NoC= lassDefFoundError: com/me/context/DefaultContext
=A0 =A0 =A0 =A0 = at java.lang.ClassLoader.defineClass1(Native Method)
=A0 =A0 =A0 = =A0 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.defineClass(ClassLoader.java:= 616)
=A0 =A0 =A0 =A0 at java.security.SecureClassLoader.defineCla= ss(SecureClassLoader.java:141)
=A0 =A0 =A0 =A0 at java.net.URLCla= ssLoader.defineClass(URLClassLoader.java:283)
=A0 =A0 =A0 =A0 at java.net.URLClassLoader.access$000(URLClassLoader.j= ava:58)
=A0 =A0 =A0 =A0 at java.net.URLClassLoader$1.run(URLClass= Loader.java:197)
=A0 =A0 =A0 =A0 at java.security.AccessControlle= r.doPrivileged(Native Method)
=A0 =A0 =A0 =A0 at java.net.URLClassLoader.findClass(URLClassLoader.ja= va:190)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.loadClass(ClassL= oader.java:307)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.loadClas= s(ClassLoader.java:248)
=A0 =A0 =A0 =A0 at com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDa= taCruncher.java:50)
=A0 =A0 =A0 =A0 at org.apache.hadoop.util.Too= lRunner.run(ToolRunner.java:65)
=A0 =A0 =A0 =A0 at org.apache.had= oop.util.ToolRunner.run(ToolRunner.java:79)
=A0 =A0 =A0 =A0 at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)<= /div>
=A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(N= ative Method)
=A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAccessor= Impl.invoke(NativeMethodAccessorImpl.java:39)
=A0 =A0 =A0 =A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del= egatingMethodAccessorImpl.java:25)
=A0 =A0 =A0 =A0 at java.lang.r= eflect.Method.invoke(Method.java:597)
=A0 =A0 =A0 =A0 at org.apac= he.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultCon= text
=A0 =A0 =A0 =A0 at java.net.URLClassLoader$1.run(URLClassLoa= der.java:202)
=A0 =A0 =A0 =A0 at java.security.AccessController.d= oPrivileged(Native Method)
=A0 =A0 =A0 =A0 at java.net.URLClassLoader.findClass(URLClassLoader.ja= va:190)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.loadClass(ClassL= oader.java:307)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.loadClas= s(ClassLoader.java:248)
--047d7b67893a2b287904d75a26f3--