Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 32292100A7 for ; Thu, 17 Apr 2014 01:47:50 +0000 (UTC) Received: (qmail 64472 invoked by uid 500); 17 Apr 2014 01:47:41 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 64331 invoked by uid 500); 17 Apr 2014 01:47:39 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 64324 invoked by uid 99); 17 Apr 2014 01:47:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 01:47:39 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of smart.rahul.iiit@gmail.com designates 209.85.212.181 as permitted sender) Received: from [209.85.212.181] (HELO mail-wi0-f181.google.com) (209.85.212.181) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 01:47:35 +0000 Received: by mail-wi0-f181.google.com with SMTP id hm4so147937wib.2 for ; Wed, 16 Apr 2014 18:47:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=lrHps6ljw1tYOiUagEdvJuTSHpwI1vOODjW4qj2dKvQ=; b=dHE7IPTjxEBiwmMZOmLCj+zVGy1K+snk1KIUZ6cUWhXC2Wt90HERbByhML2cQXL6Bo WYC5Dz1s06p7v/n2kan7CkebWMY5lZdlUpyo82ik3ylNlTCtjbCW8uxcvEixZX+S2ANH /UiGqIQDtB063RwXv9JyBXiqRvipkcArCwkrHgkQlMkgO2QQP6i8/4zowYOyFQTyS+8x DiMTwr1481bDgVdXM/0gscOBiBBMfMjimYTL7Y13udHEcCVkwmip3WMdVXQlze+Rr7/i 6aXjBPTg/hsR2P56XRN2Xscubr0ofIC13aPZipb3dr2Z1iNn7CO4hQUCiY0H1FbS0T5m onDQ== MIME-Version: 1.0 X-Received: by 10.180.106.132 with SMTP id gu4mr9257869wib.26.1397699233049; Wed, 16 Apr 2014 18:47:13 -0700 (PDT) Received: by 10.227.94.6 with HTTP; Wed, 16 Apr 2014 18:47:12 -0700 (PDT) In-Reply-To: References: <00BBFD45-D11F-4D64-ABB5-E0B44F6F13B0@hortonworks.com> Date: Thu, 17 Apr 2014 07:17:12 +0530 Message-ID: Subject: Re: using "-libjars" in Hadoop 2.2.1 From: Rahul Singh To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e89a8f3babd349f68004f73336ae X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f3babd349f68004f73336ae Content-Type: text/plain; charset=UTF-8 Please could anyone respond to my query above: Why i am getting this warning? 14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. Because of this my libjar is not getting picked up and i am getting class def not found error. Thanks and Regards, Rahul Singh On Thu, Apr 17, 2014 at 2:08 AM, Kim Chew wrote: > Thanks Rahman. This problem can be boiled down to how to submit a job > compiled with Hadoop-1.1.1 remotely to a Hadoop 2 cluster that has not > turned on YARN. I will open another thread for it. > > Kim > > > On Wed, Apr 16, 2014 at 1:30 PM, Abdelrahman Shettia < > ashettia@hortonworks.com> wrote: > >> Hi Kim, >> >> You can try to grep on the RM java process by running the following >> command: >> >> ps aux | grep >> >> >> >> >> On Wed, Apr 16, 2014 at 10:31 AM, Kim Chew wrote: >> >>> Thanks Rahman, I have mixed things up a little bit in my mapred-site.xml >>> so it tried to run the job locally. Now I am running into the problem that >>> Rahul has, I am unable to to connect to the ResourceManager. >>> >>> The setup of my targeted cluster runs MR1 instead of YARN, hence the " >>> mapreduce.framework.name" is set to "classic". >>> >>> Here are my settings in my mapred-site.xml on the client side. >>> >>> >>> >>> mapreduce.job.tracker.address >>> 172.31.3.150:8021 >>> >>> >>> mapreduce.framework.name >>> yarn >>> >>> >>> and my yarn-site.xml >>> >>> >>> The hostname of the RM. >>> yarn.resourcemanager.hostname >>> 172.31.3.150 >>> >>> >>> >>> The address of the applications manager >>> interface in the RM. >>> yarn.resourcemanager.address >>> ${yarn.resourcemanager.hostname}:8032 >>> >>> >>> 14/04/16 10:23:02 INFO client.RMProxy: Connecting to ResourceManager at / >>> 172.31.3.150:8032 >>> 14/04/16 10:23:10 INFO ipc.Client: Retrying connect to server: >>> hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); >>> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, >>> sleepTime=1 SECONDS) >>> >>> Therefore, the question is how do I figure out where the ResourceManager >>> is running? >>> >>> TIA >>> >>> Kim >>> >>> >>> >>> On Wed, Apr 16, 2014 at 8:43 AM, Abdelrahman Shettia < >>> ashettia@hortonworks.com> wrote: >>> >>>> Hi Kim, >>>> >>>> It looks like it is pointing to hdfs location. Can you create the hdfs >>>> dir and put the jar there? Hope this helps >>>> Thanks, >>>> Rahman >>>> >>>> On Apr 16, 2014, at 8:39 AM, Rahul Singh >>>> wrote: >>>> >>>> any help...all are welcome? >>>> >>>> >>>> On Wed, Apr 16, 2014 at 1:13 PM, Rahul Singh < >>>> smart.rahul.iiit@gmail.com> wrote: >>>> >>>>> Hi, >>>>> I am running with the following command but still, jar is not >>>>> available to mapper and reducers. >>>>> >>>>> hadoop jar /home/hduser/workspace/Minerva.jar my.search.Minerva >>>>> /user/hduser/input_minerva_actual /user/hduser/output_merva_actual3 >>>>> -libjars /home/hduser/Documents/Lib/json-simple-1.1.1.jar >>>>> -Dmapreduce.user.classpath.first=true >>>>> >>>>> >>>>> Error Log >>>>> >>>>> 14/04/16 13:08:37 INFO client.RMProxy: Connecting to ResourceManager >>>>> at /0.0.0.0:8032 >>>>> 14/04/16 13:08:37 INFO client.RMProxy: Connecting to ResourceManager >>>>> at /0.0.0.0:8032 >>>>> 14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line >>>>> option parsing not performed. Implement the Tool interface and execute your >>>>> application with ToolRunner to remedy this. >>>>> 14/04/16 13:08:37 INFO mapred.FileInputFormat: Total input paths to >>>>> process : 1 >>>>> 14/04/16 13:08:37 INFO mapreduce.JobSubmitter: number of splits:10 >>>>> 14/04/16 13:08:37 INFO mapreduce.JobSubmitter: Submitting tokens for >>>>> job: job_1397534064728_0028 >>>>> 14/04/16 13:08:38 INFO impl.YarnClientImpl: Submitted application >>>>> application_1397534064728_0028 >>>>> 14/04/16 13:08:38 INFO mapreduce.Job: The url to track the job: >>>>> http://L-Rahul-Tech:8088/proxy/application_1397534064728_0028/ >>>>> 14/04/16 13:08:38 INFO mapreduce.Job: Running job: >>>>> job_1397534064728_0028 >>>>> 14/04/16 13:08:47 INFO mapreduce.Job: Job job_1397534064728_0028 >>>>> running in uber mode : false >>>>> 14/04/16 13:08:47 INFO mapreduce.Job: map 0% reduce 0% >>>>> 14/04/16 13:08:58 INFO mapreduce.Job: Task Id : >>>>> attempt_1397534064728_0028_m_000005_0, Status : FAILED >>>>> Error: java.lang.RuntimeException: Error in configuring object >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) >>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) >>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >>>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>> at >>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) >>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) >>>>> Caused by: java.lang.reflect.InvocationTargetException >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:622) >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) >>>>> ... 9 more >>>>> Caused by: java.lang.NoClassDefFoundError: >>>>> org/json/simple/parser/ParseException >>>>> at java.lang.Class.forName0(Native Method) >>>>> at java.lang.Class.forName(Class.java:270) >>>>> at >>>>> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1821) >>>>> at >>>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1786) >>>>> at >>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1880) >>>>> at >>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1906) >>>>> at >>>>> org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1107) >>>>> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) >>>>> ... 14 more >>>>> Caused by: java.lang.ClassNotFoundException: >>>>> org.json.simple.parser.ParseException >>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:323) >>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:268) >>>>> ... 22 more >>>>> >>>>> When i analyzed the logs it says >>>>> "14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line >>>>> option parsing not performed. Implement the Tool interface and execute your >>>>> application with ToolRunner to remedy this." >>>>> >>>>> But i have implemented the tool class as described below: >>>>> >>>>> package my.search; >>>>> >>>>> import org.apache.hadoop.conf.Configured; >>>>> import org.apache.hadoop.fs.Path; >>>>> import org.apache.hadoop.io.Text; >>>>> import org.apache.hadoop.mapred.FileInputFormat; >>>>> import org.apache.hadoop.mapred.FileOutputFormat; >>>>> import org.apache.hadoop.mapred.JobClient; >>>>> import org.apache.hadoop.mapred.JobConf; >>>>> import org.apache.hadoop.mapred.TextInputFormat; >>>>> import org.apache.hadoop.mapred.TextOutputFormat; >>>>> import org.apache.hadoop.util.Tool; >>>>> import org.apache.hadoop.util.ToolRunner; >>>>> >>>>> public class Minerva extends Configured implements Tool >>>>> { >>>>> public int run(String[] args) throws Exception { >>>>> JobConf conf = new JobConf(Minerva.class); >>>>> conf.setJobName("minerva sample job"); >>>>> >>>>> conf.setMapOutputKeyClass(Text.class); >>>>> conf.setMapOutputValueClass(TextArrayWritable.class); >>>>> >>>>> conf.setOutputKeyClass(Text.class); >>>>> conf.setOutputValueClass(Text.class); >>>>> >>>>> conf.setMapperClass(Map.class); >>>>> // conf.setCombinerClass(Reduce.class); >>>>> conf.setReducerClass(Reduce.class); >>>>> >>>>> conf.setInputFormat(TextInputFormat.class); >>>>> conf.setOutputFormat(TextOutputFormat.class); >>>>> >>>>> FileInputFormat.setInputPaths(conf, new Path(args[0])); >>>>> FileOutputFormat.setOutputPath(conf, new Path(args[1])); >>>>> >>>>> JobClient.runJob(conf); >>>>> >>>>> return 0; >>>>> } >>>>> >>>>> public static void main(String[] args) throws Exception { >>>>> int res = ToolRunner.run(new Minerva(), args); >>>>> System.exit(res); >>>>> } >>>>> } >>>>> >>>>> >>>>> Please let me know if you see any issues? >>>>> >>>>> >>>>> >>>>> On Thu, Apr 10, 2014 at 9:29 AM, Shengjun Xin wrote: >>>>> >>>>>> add '-Dmapreduce.user.classpath.first=true' to your command and try >>>>>> again >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 9, 2014 at 6:27 AM, Kim Chew wrote: >>>>>> >>>>>>> It seems to me that in Hadoop 2.2.1, using the "libjars" option does >>>>>>> not search the jars located in the the local file system but HDFS. For >>>>>>> example, >>>>>>> >>>>>>> hadoop jar target/myJar.jar Foo -libjars >>>>>>> /home/kchew/test-libs/testJar.jar /user/kchew/inputs/raw.vector >>>>>>> /user/kchew/outputs hdfs://remoteNN:8020 remoteJT:8021 >>>>>>> >>>>>>> 14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with >>>>>>> processName=JobTracker, sessionId= >>>>>>> 14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the >>>>>>> staging area >>>>>>> file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001 >>>>>>> 14/04/08 15:11:02 ERROR security.UserGroupInformation: >>>>>>> PriviledgedActionException as:kchew (auth:SIMPLE) >>>>>>> cause:java.io.FileNotFoundException: File does not exist: >>>>>>> hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar >>>>>>> java.io.FileNotFoundException: File does not exist: >>>>>>> hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar >>>>>>> at >>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110) >>>>>>> at >>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102) >>>>>>> at >>>>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >>>>>>> at >>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102) >>>>>>> at >>>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288) >>>>>>> at >>>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224) >>>>>>> at >>>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93) >>>>>>> at >>>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57) >>>>>>> at >>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264) >>>>>>> >>>>>>> So under Hadoop 2.2.1, do I have to explicitly set some >>>>>>> configurations so when using the "libjars" option it will copy the file to >>>>>>> hdfs from local fs? >>>>>>> >>>>>>> TIA >>>>>>> >>>>>>> Kim >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards >>>>>> Shengjun >>>>>> >>>>> >>>>> >>>> >>>> >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to which it is addressed and may contain information that is >>>> confidential, privileged and exempt from disclosure under applicable law. >>>> If the reader of this message is not the intended recipient, you are hereby >>>> notified that any printing, copying, dissemination, distribution, >>>> disclosure or forwarding of this communication is strictly prohibited. If >>>> you have received this communication in error, please contact the sender >>>> immediately and delete it from your system. Thank You. >>> >>> >>> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > > --e89a8f3babd349f68004f73336ae Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Please could anyone respond to my query above:

Why i am getting this warning?

14/04/16 13:08:37 WARN mapreduce= .JobSubmitter: Hadoop command-line=20 option parsing not performed. Implement the Tool interface and execute=20 your application with ToolRunner to remedy this.

Because = of this my libjar is not getting picked up and i am getting class def not f= ound error.

Thanks and Regards,
Rahul Singh=


O= n Thu, Apr 17, 2014 at 2:08 AM, Kim Chew <kchew534@gmail.com> wrote:
Thanks Rahman. This pr= oblem can be boiled down to how to submit a job compiled with Hadoop-1.1.1 = remotely to a Hadoop 2 cluster that has not turned on YARN. I will open ano= ther thread for it.

Kim


On Wed, Apr 16, 2014 = at 1:30 PM, Abdelrahman Shettia <ashettia@hortonworks.com> wrote:
Hi Kim,=C2=A0

You can try to grep on the RM java process by running the following c= ommand:=C2=A0

ps aux | grep=C2=A0


=


On Wed, Apr 16, 2014 at 10:31 AM, Kim Ch= ew <kchew534@gmail.com> wrote:
Thanks Rahman, I have mixed = things up a little bit in my mapred-site.xml so it tried to run the job loc= ally. Now I am running into the problem that Rahul has, I am unable to to c= onnect to the ResourceManager.

The setup of my targeted cluster runs MR1 instead of YARN, hence = the "
mapreduce.framework.name" is set to "classic".

Here are my settings in my mapred-site.xml on the client side.

<property>
=C2=A0=C2=A0=C2=A0 <!-- Pointed to the remote Jo= bTracker -->
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <name>m= apreduce.job.tracker.address</name>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 <value>172.31.3.150:8021</value>
=C2=A0=C2=A0=C2=A0 </property>
=C2=A0=C2=A0=C2=A0 <property>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <name>mapreduce.framework.name&= lt;/name>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <value>yar= n</value>
=C2=A0=C2=A0=C2=A0 </property>

and my yarn-site.xml

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = <property>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 <description>The hostname of the RM.</description>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <= name>yarn.resourcemanager.hostname</name>
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <value>172.31.3.150<= /value>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </property>=C2=A0=C2=A0=C2= =A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <prope= rty>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 <description>The address of the applications manager interface in= the RM.</description>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 <name>yarn.resourcemanager.address</name&= gt;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <valu= e>${yarn.resourcemanager.hostname}:8032</value>
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 </property>

14/04/16 10:23:02 INFO= client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
14/04/16 10:23:10 INFO ipc.Client: Retrying connect to server: hadoo= p-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry= policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sleepTime= =3D1 SECONDS)

Therefore, the question is how do I figure out where the Resource= Manager is running?

TIA

Kim


On Wed, Apr 16, 2014 at 8:43 AM, Abdelrahman Shettia <= ashettia@hort= onworks.com> wrote:
Hi Kim,

It looks lik= e it is pointing to hdfs location. Can you create the hdfs dir and put the = jar there? Hope this helps=C2=A0
Thanks,
Rahman

On Apr 16, 2014, at 8:39 AM, Rahul Singh <smart.rahul.iiit@gmail.com<= /a>> wrote:

any help= ...all are welcome?


On Wed,= Apr 16, 2014 at 1:13 PM, Rahul Singh <smart.rahul.iiit@gmail.com= > wrote:
Hi,
=C2= =A0I am running with the following command but still, jar is not available = to mapper and reducers.

hadoop jar /home/hduser/workspace/Minerva.jar my.search.Minerva /user/h= duser/input_minerva_actual /user/hduser/output_merva_actual3 -libjars /home= /hduser/Documents/Lib/json-simple-1.1.1.jar -Dmapreduce.user.classpath.firs= t=3Dtrue


Error Log

14/04/16 13:08:37 INFO client.RMProxy: Conne= cting to ResourceManager at /0.0.0.0:8032
14/04/16 13:08:37 INFO client.RMProxy: Connecting = to ResourceManager at /0= .0.0.0:8032
14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line option p= arsing not performed. Implement the Tool interface and execute your applica= tion with ToolRunner to remedy this.
14/04/16 13:08:37 INFO mapred.FileI= nputFormat: Total input paths to process : 1
14/04/16 13:08:37 INFO mapreduce.JobSubmitter: number of splits:10
14/04= /16 13:08:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_13= 97534064728_0028
14/04/16 13:08:38 INFO impl.YarnClientImpl: Submitted a= pplication application_1397534064728_0028
14/04/16 13:08:38 INFO mapreduce.Job: The url to track the job: http://L-Rahul-Tech:8088/proxy/application_1397534064728_0028/ 14/04/16 13:08:38 INFO mapreduce.Job: Running job: job_1397534064728_0028 14/04/16 13:08:47 INFO mapreduce.Job: Job job_1397534064728_0028 running in= uber mode : false
14/04/16 13:08:47 INFO mapreduce.Job:=C2=A0 map 0% re= duce 0%
14/04/16 13:08:58 INFO mapreduce.Job: Task Id : attempt_13975340= 64728_0028_m_000005_0, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
=C2=A0=C2= =A0=C2=A0 at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUt= ils.java:109)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.ReflectionUti= ls.setConf(ReflectionUtils.java:75)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.ReflectionUtils.newInstance(Re= flectionUtils.java:133)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapred.M= apTask.runOldMapper(MapTask.java:426)
=C2=A0=C2=A0=C2=A0 at org.apache.h= adoop.mapred.MapTask.run(MapTask.java:342)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.ja= va:168)
=C2=A0=C2=A0=C2=A0 at java.security.AccessController.doPrivilege= d(Native Method)
=C2=A0=C2=A0=C2=A0 at javax.security.auth.Subject.doAs(= Subject.java:416)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.security.UserG= roupInformation.doAs(UserGroupInformation.java:1548)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.jav= a:163)
Caused by: java.lang.reflect.InvocationTargetException
=C2=A0= =C2=A0=C2=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)=
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(Nativ= eMethodAccessorImpl.java:57)
=C2=A0=C2=A0=C2=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Deleg= atingMethodAccessorImpl.java:43)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect= .Method.invoke(Method.java:622)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.= util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
=C2=A0=C2=A0=C2=A0 ... 9 more
Caused by: java.lang.NoClassDefFoundError:= org/json/simple/parser/ParseException
=C2=A0=C2=A0=C2=A0 at java.lang.C= lass.forName0(Native Method)
=C2=A0=C2=A0=C2=A0 at java.lang.Class.forNa= me(Class.java:270)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configur= ation.getClassByNameOrNull(Configuration.java:1821)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configuration.getClassByName(C= onfiguration.java:1786)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Con= figuration.getClass(Configuration.java:1880)
=C2=A0=C2=A0=C2=A0 at org.a= pache.hadoop.conf.Configuration.getClass(Configuration.java:1906)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapred.JobConf.getMapperClass(JobCo= nf.java:1107)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapred.MapRunner.c= onfigure(MapRunner.java:38)
=C2=A0=C2=A0=C2=A0 ... 14 more
Caused by:= java.lang.ClassNotFoundException: org.json.simple.parser.ParseException =C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:217= )
=C2=A0=C2=A0=C2=A0 at java.security.AccessController.doPrivileged(Nati= ve Method)
=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader.findClass(URLCl= assLoader.java:205)
=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoader.loadClas= s(ClassLoader.java:323)
=C2=A0=C2=A0=C2=A0 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.j= ava:294)
=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoader.loadClass(ClassLoad= er.java:268)
=C2=A0=C2=A0=C2=A0 ... 22 more

When i analyzed= the logs it says
"14/04/16 13:08:37 WARN mapreduce.JobSubmitter: H= adoop command-line option parsing not performed. Implement the Tool interfa= ce and execute your application with ToolRunner to remedy this."

But i have implemented the tool class as described below:
package my.search;

import org.apache.hadoop.conf.Configured;
imp= ort org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoo= p.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapre= d.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hado= op.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public cl= ass Minerva extends Configured implements Tool
{
=C2=A0=C2=A0=C2=A0 p= ublic int run(String[] args) throws Exception {
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 JobConf conf =3D new JobConf(Minerva.= class);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setJobName("mine= rva sample job");

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.se= tMapOutputKeyClass(Text.class);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 co= nf.setMapOutputValueClass(TextArrayWritable.class);

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setOutputKeyClass(Text.class= );
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setOutputValueClass(Text.c= lass);

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setMapperClass(Map= .class);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 // conf.setCombinerClass(= Reduce.class);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setReducerClas= s(Reduce.class);

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setInputFormat(TextInputForm= at.class);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 conf.setOutputFormat(Te= xtOutputFormat.class);

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 FileInp= utFormat.setInputPaths(conf, new Path(args[0]));
=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 FileOutputFormat.setOutputPath(conf, new Path(args[1]));
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 JobClient.runJob(conf);
=C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 r= eturn 0;
=C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2= =A0 public static void main(String[] args) throws Exception {
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 int res =3D ToolRunner.run(new Minerva(),= args);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 System.exit(res);
=C2=A0=C2=A0=C2=A0 }
}


Please let me know if you see an= y issues?



On Thu, Apr 10, 2014 at 9:29 AM, Shengjun Xin <sxin@g= opivotal.com> wrote:
add '-Dmapreduce.user.c= lasspath.first=3Dtrue' to your command and try again



On Wed, Apr 9, 2014 at 6:27 AM, Kim= Chew <kchew534@gmail.com> wrote:
It seems to me that in Had= oop 2.2.1, using the "libjars" option does not search the jars lo= cated in the the local file system but HDFS. For example,

hadoop jar target/myJar.jar Foo -libjars /home/kchew/test-libs/testJar.= jar /user/kchew/inputs/raw.vector /user/kchew/outputs hdfs://remoteNN:80= 20 remoteJT:8021

14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with pr= ocessName=3DJobTracker, sessionId=3D
14/04/08 15:11:02 INFO mapreduce.Jo= bSubmitter: Cleaning up the staging area file:/tmp/hadoop-kchew/mapred/stag= ing/kchew202924688/.staging/job_local202924688_0001
14/04/08 15:11:02 ERROR security.UserGroupInformation: PriviledgedActionExc= eption as:kchew (auth:SIMPLE) cause:java.io.FileNotFoundException: File doe= s not exist: hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar java.io.FileNotFoundException: File does not exist: hdfs:/remoteNN:8020/hom= e/kchew/test-libs/testJar.jar
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCal= l(DistributedFileSystem.java:1110)
=C2=A0=C2=A0=C2=A0 at org.apache.hado= op.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(= FileSystemLinkResolver.java:81)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileS= tatus(DistributedFileSystem.java:1102)
=C2=A0=C2=A0=C2=A0 at org.apache.= hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(Clie= ntDistributedCacheManager.java:288)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.filecache.ClientDistribut= edCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
=C2= =A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.filecache.ClientDistributedC= acheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.filecache.ClientDistribut= edCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCac= heManager.java:57)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.Job= Submitter.copyAndConfigureFiles(JobSubmitter.java:264)

So under Hadoop 2.2.1, do I have to explicitly set some configura= tions so when using the "libjars" option it will copy the file to= hdfs from local fs?

TIA

Kim
=



--
Regards
Shengjun




CONFIDENTIALITY NOTICE
NOTICE: This= message is intended for the use of the individual or entity to which it is= addressed and may contain information that is confidential, privileged and= exempt from disclosure under applicable law. If the reader of this message= is not the intended recipient, you are hereby notified that any printing, = copying, dissemination, distribution, disclosure or forwarding of this comm= unication is strictly prohibited. If you have received this communication i= n error, please contact the sender immediately and delete it from your syst= em. Thank You.



CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.


--e89a8f3babd349f68004f73336ae--