Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1664610443 for ; Wed, 16 Apr 2014 20:31:28 +0000 (UTC) Received: (qmail 94827 invoked by uid 500); 16 Apr 2014 20:31:20 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 94717 invoked by uid 500); 16 Apr 2014 20:31:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 94710 invoked by uid 99); 16 Apr 2014 20:31:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2014 20:31:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ashettia@hortonworks.com designates 209.85.219.48 as permitted sender) Received: from [209.85.219.48] (HELO mail-oa0-f48.google.com) (209.85.219.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2014 20:31:16 +0000 Received: by mail-oa0-f48.google.com with SMTP id m1so13020483oag.35 for ; Wed, 16 Apr 2014 13:30:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=FOT8POh+t/C5DocRe3iFIyrhyId7z1a8kqtCwVi9cAg=; b=Bi0Of7+h+pMs93a2PfnKJQrsSNJQ1A8jurN4rNxV6cvVQXcrCYZNcNmVd41XtDyhvs 7+uUQzgFmiyWnRudvg4hsbsGfMjWp3CQsK21QKWRvBhuelkOoOv5o99vksqZMM1gUqOq LXPD+cjEv/d2UUgw2NudnWgqDWrUY/Ory5kuqHVze04uzb/ARvQcPwTueL9HjzDcnKNY MepFbgvKSu89GUUcl06s/QHQFPAPAYspnQeSltRaxGSCggMx1gOIE22Z6x7sq2r7Ea2q uqRnnNZi0/tOfUno8yEhmD7pgu2n4sPaYkFX1nHDttZbHWZTGIhHNiU+u74uPNvw+xLy Dp5Q== X-Gm-Message-State: ALoCoQkdUma4/Mjt/quvLImKBL8HHxFfzyaithDEsi8rSmKnMPRLq2wQb0OPvWA5nLb8dMRYkJbkt5Ucg87ChGzD9kFPxGENFBrAI1CiCYuL7lLQznGObOg= MIME-Version: 1.0 X-Received: by 10.60.16.103 with SMTP id f7mr8343606oed.8.1397680255113; Wed, 16 Apr 2014 13:30:55 -0700 (PDT) Received: by 10.182.24.169 with HTTP; Wed, 16 Apr 2014 13:30:55 -0700 (PDT) In-Reply-To: References: <00BBFD45-D11F-4D64-ABB5-E0B44F6F13B0@hortonworks.com> Date: Wed, 16 Apr 2014 13:30:55 -0700 Message-ID: Subject: Re: using "-libjars" in Hadoop 2.2.1 From: Abdelrahman Shettia To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0149c04e1dc63604f72ecbb0 X-Virus-Checked: Checked by ClamAV on apache.org --089e0149c04e1dc63604f72ecbb0 Content-Type: text/plain; charset=US-ASCII Hi Kim, You can try to grep on the RM java process by running the following command: ps aux | grep On Wed, Apr 16, 2014 at 10:31 AM, Kim Chew wrote: > Thanks Rahman, I have mixed things up a little bit in my mapred-site.xml > so it tried to run the job locally. Now I am running into the problem that > Rahul has, I am unable to to connect to the ResourceManager. > > The setup of my targeted cluster runs MR1 instead of YARN, hence the " > mapreduce.framework.name" is set to "classic". > > Here are my settings in my mapred-site.xml on the client side. > > > > mapreduce.job.tracker.address > 172.31.3.150:8021 > > > mapreduce.framework.name > yarn > > > and my yarn-site.xml > > > The hostname of the RM. > yarn.resourcemanager.hostname > 172.31.3.150 > > > > The address of the applications manager interface > in the RM. > yarn.resourcemanager.address > ${yarn.resourcemanager.hostname}:8032 > > > 14/04/16 10:23:02 INFO client.RMProxy: Connecting to ResourceManager at / > 172.31.3.150:8032 > 14/04/16 10:23:10 INFO ipc.Client: Retrying connect to server: > hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); > retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, > sleepTime=1 SECONDS) > > Therefore, the question is how do I figure out where the ResourceManager > is running? > > TIA > > Kim > > > > On Wed, Apr 16, 2014 at 8:43 AM, Abdelrahman Shettia < > ashettia@hortonworks.com> wrote: > >> Hi Kim, >> >> It looks like it is pointing to hdfs location. Can you create the hdfs >> dir and put the jar there? Hope this helps >> Thanks, >> Rahman >> >> On Apr 16, 2014, at 8:39 AM, Rahul Singh >> wrote: >> >> any help...all are welcome? >> >> >> On Wed, Apr 16, 2014 at 1:13 PM, Rahul Singh wrote: >> >>> Hi, >>> I am running with the following command but still, jar is not available >>> to mapper and reducers. >>> >>> hadoop jar /home/hduser/workspace/Minerva.jar my.search.Minerva >>> /user/hduser/input_minerva_actual /user/hduser/output_merva_actual3 >>> -libjars /home/hduser/Documents/Lib/json-simple-1.1.1.jar >>> -Dmapreduce.user.classpath.first=true >>> >>> >>> Error Log >>> >>> 14/04/16 13:08:37 INFO client.RMProxy: Connecting to ResourceManager at / >>> 0.0.0.0:8032 >>> 14/04/16 13:08:37 INFO client.RMProxy: Connecting to ResourceManager at / >>> 0.0.0.0:8032 >>> 14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line >>> option parsing not performed. Implement the Tool interface and execute your >>> application with ToolRunner to remedy this. >>> 14/04/16 13:08:37 INFO mapred.FileInputFormat: Total input paths to >>> process : 1 >>> 14/04/16 13:08:37 INFO mapreduce.JobSubmitter: number of splits:10 >>> 14/04/16 13:08:37 INFO mapreduce.JobSubmitter: Submitting tokens for >>> job: job_1397534064728_0028 >>> 14/04/16 13:08:38 INFO impl.YarnClientImpl: Submitted application >>> application_1397534064728_0028 >>> 14/04/16 13:08:38 INFO mapreduce.Job: The url to track the job: >>> http://L-Rahul-Tech:8088/proxy/application_1397534064728_0028/ >>> 14/04/16 13:08:38 INFO mapreduce.Job: Running job: job_1397534064728_0028 >>> 14/04/16 13:08:47 INFO mapreduce.Job: Job job_1397534064728_0028 running >>> in uber mode : false >>> 14/04/16 13:08:47 INFO mapreduce.Job: map 0% reduce 0% >>> 14/04/16 13:08:58 INFO mapreduce.Job: Task Id : >>> attempt_1397534064728_0028_m_000005_0, Status : FAILED >>> Error: java.lang.RuntimeException: Error in configuring object >>> at >>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) >>> at >>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:416) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) >>> Caused by: java.lang.reflect.InvocationTargetException >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:622) >>> at >>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) >>> ... 9 more >>> Caused by: java.lang.NoClassDefFoundError: >>> org/json/simple/parser/ParseException >>> at java.lang.Class.forName0(Native Method) >>> at java.lang.Class.forName(Class.java:270) >>> at >>> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1821) >>> at >>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1786) >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1880) >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1906) >>> at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1107) >>> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) >>> ... 14 more >>> Caused by: java.lang.ClassNotFoundException: >>> org.json.simple.parser.ParseException >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:323) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:268) >>> ... 22 more >>> >>> When i analyzed the logs it says >>> "14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line >>> option parsing not performed. Implement the Tool interface and execute your >>> application with ToolRunner to remedy this." >>> >>> But i have implemented the tool class as described below: >>> >>> package my.search; >>> >>> import org.apache.hadoop.conf.Configured; >>> import org.apache.hadoop.fs.Path; >>> import org.apache.hadoop.io.Text; >>> import org.apache.hadoop.mapred.FileInputFormat; >>> import org.apache.hadoop.mapred.FileOutputFormat; >>> import org.apache.hadoop.mapred.JobClient; >>> import org.apache.hadoop.mapred.JobConf; >>> import org.apache.hadoop.mapred.TextInputFormat; >>> import org.apache.hadoop.mapred.TextOutputFormat; >>> import org.apache.hadoop.util.Tool; >>> import org.apache.hadoop.util.ToolRunner; >>> >>> public class Minerva extends Configured implements Tool >>> { >>> public int run(String[] args) throws Exception { >>> JobConf conf = new JobConf(Minerva.class); >>> conf.setJobName("minerva sample job"); >>> >>> conf.setMapOutputKeyClass(Text.class); >>> conf.setMapOutputValueClass(TextArrayWritable.class); >>> >>> conf.setOutputKeyClass(Text.class); >>> conf.setOutputValueClass(Text.class); >>> >>> conf.setMapperClass(Map.class); >>> // conf.setCombinerClass(Reduce.class); >>> conf.setReducerClass(Reduce.class); >>> >>> conf.setInputFormat(TextInputFormat.class); >>> conf.setOutputFormat(TextOutputFormat.class); >>> >>> FileInputFormat.setInputPaths(conf, new Path(args[0])); >>> FileOutputFormat.setOutputPath(conf, new Path(args[1])); >>> >>> JobClient.runJob(conf); >>> >>> return 0; >>> } >>> >>> public static void main(String[] args) throws Exception { >>> int res = ToolRunner.run(new Minerva(), args); >>> System.exit(res); >>> } >>> } >>> >>> >>> Please let me know if you see any issues? >>> >>> >>> >>> On Thu, Apr 10, 2014 at 9:29 AM, Shengjun Xin wrote: >>> >>>> add '-Dmapreduce.user.classpath.first=true' to your command and try >>>> again >>>> >>>> >>>> >>>> On Wed, Apr 9, 2014 at 6:27 AM, Kim Chew wrote: >>>> >>>>> It seems to me that in Hadoop 2.2.1, using the "libjars" option does >>>>> not search the jars located in the the local file system but HDFS. For >>>>> example, >>>>> >>>>> hadoop jar target/myJar.jar Foo -libjars >>>>> /home/kchew/test-libs/testJar.jar /user/kchew/inputs/raw.vector >>>>> /user/kchew/outputs hdfs://remoteNN:8020 remoteJT:8021 >>>>> >>>>> 14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with >>>>> processName=JobTracker, sessionId= >>>>> 14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the staging >>>>> area >>>>> file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001 >>>>> 14/04/08 15:11:02 ERROR security.UserGroupInformation: >>>>> PriviledgedActionException as:kchew (auth:SIMPLE) >>>>> cause:java.io.FileNotFoundException: File does not exist: >>>>> hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar >>>>> java.io.FileNotFoundException: File does not exist: >>>>> hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar >>>>> at >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110) >>>>> at >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102) >>>>> at >>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >>>>> at >>>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102) >>>>> at >>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288) >>>>> at >>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224) >>>>> at >>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93) >>>>> at >>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57) >>>>> at >>>>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264) >>>>> >>>>> So under Hadoop 2.2.1, do I have to explicitly set some configurations >>>>> so when using the "libjars" option it will copy the file to hdfs from local >>>>> fs? >>>>> >>>>> TIA >>>>> >>>>> Kim >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards >>>> Shengjun >>>> >>> >>> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --089e0149c04e1dc63604f72ecbb0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Kim,=A0

You can try to grep on the R= M java process by running the following command:=A0

ps aux | grep=A0




On Wed, Apr 16, 2014 at 10:31 AM, Kim Ch= ew <kchew534@gmail.com> wrote:
Thanks Rahman, I have mixed = things up a little bit in my mapred-site.xml so it tried to run the job loc= ally. Now I am running into the problem that Rahul has, I am unable to to c= onnect to the ResourceManager.

The setup of my targeted cluster runs MR1 instead of YARN, hence = the "
mapreduce.framework.name" is set to "classic".

Here are my settings in my mapred-site.xml on the client side.

<property>
=A0=A0=A0 <!-- Pointed to the remote JobTracker = -->
=A0=A0=A0=A0=A0=A0=A0 <name>mapreduce.job.tracker.address&l= t;/name>
=A0=A0=A0=A0=A0=A0=A0 <value>172.31.3.150:8021</value>
=A0=A0=A0 </property>
=A0=A0=A0 <property>
=A0=A0=A0=A0= =A0=A0=A0 <name>mapreduce.framework.name</name>
=A0=A0=A0=A0=A0=A0=A0= <value>yarn</value>
=A0=A0=A0 </property>

and my yarn-site.xml

=A0=A0=A0=A0=A0=A0 <property>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <description>The hostname of the = RM.</description>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <name>ya= rn.resourcemanager.hostname</name>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 <value>172.31.3.150</value>
=A0=A0=A0=A0=A0=A0=A0 </property>=A0=A0=A0
=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0
=A0=A0=A0=A0=A0=A0=A0 <property>
=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 <description>The address of the applicati= ons manager interface in the RM.</description>
=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 <name>yarn.resourcemanager.address</name>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <value>${yarn.resourcemanager.hostn= ame}:8032</value>
=A0=A0=A0=A0=A0=A0=A0 </property>

1= 4/04/16 10:23:02 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032 14/04/16 10:23:10 INFO ipc.Client: Retrying connect to server: hadoo= p-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry= policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sleepTime= =3D1 SECONDS)

Therefore, the question is how do I figure out where the Resource= Manager is running?

TIA

Kim


On Wed, Apr 16, 2014 at 8:43 AM, Abdelrahman Shettia <= ashettia@hort= onworks.com> wrote:
Hi Kim,

It looks lik= e it is pointing to hdfs location. Can you create the hdfs dir and put the = jar there? Hope this helps=A0
Thanks,
Rahman

On Apr 16, 2014, at 8:39 AM, Rahul Singh <smart.rahul.iiit@gmail.com<= /a>> wrote:

any help= ...all are welcome?


On Wed,= Apr 16, 2014 at 1:13 PM, Rahul Singh <smart.rahul.iiit@gmail.com= > wrote:
Hi,
=A0= I am running with the following command but still, jar is not available to = mapper and reducers.

hadoop jar /home/hduser/workspace/Minerva.jar my.search.Minerva /user/h= duser/input_minerva_actual /user/hduser/output_merva_actual3 -libjars /home= /hduser/Documents/Lib/json-simple-1.1.1.jar -Dmapreduce.user.classpath.firs= t=3Dtrue


Error Log

14/04/16 13:08:37 INFO client.RMProxy: Conne= cting to ResourceManager at /0.0.0.0:8032
14/04/16 13:08:37 INFO client.RMProxy: Connecting = to ResourceManager at /0= .0.0.0:8032
14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line option p= arsing not performed. Implement the Tool interface and execute your applica= tion with ToolRunner to remedy this.
14/04/16 13:08:37 INFO mapred.FileI= nputFormat: Total input paths to process : 1
14/04/16 13:08:37 INFO mapreduce.JobSubmitter: number of splits:10
14/04= /16 13:08:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_13= 97534064728_0028
14/04/16 13:08:38 INFO impl.YarnClientImpl: Submitted a= pplication application_1397534064728_0028
14/04/16 13:08:38 INFO mapreduce.Job: The url to track the job: http://L-Rahul-Tech:8088/proxy/application_1397534064728_0028/ 14/04/16 13:08:38 INFO mapreduce.Job: Running job: job_1397534064728_0028 14/04/16 13:08:47 INFO mapreduce.Job: Job job_1397534064728_0028 running in= uber mode : false
14/04/16 13:08:47 INFO mapreduce.Job:=A0 map 0% reduc= e 0%
14/04/16 13:08:58 INFO mapreduce.Job: Task Id : attempt_13975340647= 28_0028_m_000005_0, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
=A0=A0=A0= at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:= 109)
=A0=A0=A0 at org.apache.hadoop.util.ReflectionUtils.setConf(Reflect= ionUtils.java:75)
=A0=A0=A0 at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionU= tils.java:133)
=A0=A0=A0 at org.apache.hadoop.mapred.MapTask.runOldMappe= r(MapTask.java:426)
=A0=A0=A0 at org.apache.hadoop.mapred.MapTask.run(Ma= pTask.java:342)
=A0=A0=A0 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)=A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:416)
=A0=A0= =A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1548)
=A0=A0=A0 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.reflect.InvocationTargetException
=A0=A0=A0 at sun= .reflect.NativeMethodAccessorImpl.invoke0(Native Method)
=A0=A0=A0 at su= n.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)=
=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43)
=A0=A0=A0 at java.lang.reflect.Method.invoke(Met= hod.java:622)
=A0=A0=A0 at org.apache.hadoop.util.ReflectionUtils.setJob= Conf(ReflectionUtils.java:106)
=A0=A0=A0 ... 9 more
Caused by: java.lang.NoClassDefFoundError: org/json= /simple/parser/ParseException
=A0=A0=A0 at java.lang.Class.forName0(Nati= ve Method)
=A0=A0=A0 at java.lang.Class.forName(Class.java:270)
=A0= =A0=A0 at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configu= ration.java:1821)
=A0=A0=A0 at org.apache.hadoop.conf.Configuration.getClassByName(Configurat= ion.java:1786)
=A0=A0=A0 at org.apache.hadoop.conf.Configuration.getClas= s(Configuration.java:1880)
=A0=A0=A0 at org.apache.hadoop.conf.Configura= tion.getClass(Configuration.java:1906)
=A0=A0=A0 at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1= 107)
=A0=A0=A0 at org.apache.hadoop.mapred.MapRunner.configure(MapRunner= .java:38)
=A0=A0=A0 ... 14 more
Caused by: java.lang.ClassNotFoundExc= eption: org.json.simple.parser.ParseException
=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
=A0= =A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0= =A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
=A0=A0=A0 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)<= br>=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
= =A0=A0=A0 ... 22 more

When i analyzed the logs it says
&quo= t;14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line option= parsing not performed. Implement the Tool interface and execute your appli= cation with ToolRunner to remedy this."

But i have implemented the tool class as described below:
package my.search;

import org.apache.hadoop.conf.Configured;
imp= ort org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoo= p.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapre= d.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hado= op.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public cl= ass Minerva extends Configured implements Tool
{
=A0=A0=A0 public int= run(String[] args) throws Exception {
=A0=A0=A0 =A0=A0=A0 JobConf conf =3D new JobConf(Minerva.class);
=A0=A0= =A0 =A0=A0=A0 conf.setJobName("minerva sample job");

=A0= =A0=A0 =A0=A0=A0 conf.setMapOutputKeyClass(Text.class);
=A0=A0=A0 =A0=A0= =A0 conf.setMapOutputValueClass(TextArrayWritable.class);

=A0=A0=A0 =A0=A0=A0 conf.setOutputKeyClass(Text.class);
=A0=A0=A0 = =A0=A0=A0 conf.setOutputValueClass(Text.class);

=A0=A0=A0 =A0=A0=A0 = conf.setMapperClass(Map.class);
=A0=A0=A0 =A0=A0=A0 // conf.setCombinerC= lass(Reduce.class);
=A0=A0=A0 =A0=A0=A0 conf.setReducerClass(Reduce.clas= s);

=A0=A0=A0 =A0=A0=A0 conf.setInputFormat(TextInputFormat.class);
=A0= =A0=A0 =A0=A0=A0 conf.setOutputFormat(TextOutputFormat.class);

=A0= =A0=A0 =A0=A0=A0 FileInputFormat.setInputPaths(conf, new Path(args[0]));=A0=A0=A0 =A0=A0=A0 FileOutputFormat.setOutputPath(conf, new Path(args[1])= );

=A0=A0=A0 =A0=A0=A0 JobClient.runJob(conf);
=A0=A0=A0 =A0=A0=A0
= =A0=A0=A0 =A0=A0=A0 return 0;
=A0=A0=A0 }
=A0=A0=A0
=A0=A0=A0 pub= lic static void main(String[] args) throws Exception {
=A0=A0=A0=A0=A0= =A0=A0 int res =3D ToolRunner.run(new Minerva(), args);
=A0=A0=A0=A0=A0= =A0=A0 System.exit(res);
=A0=A0=A0 }
}


Please let me know if you see any issues?=



On Thu, Apr 10, 2014 at 9:29 AM, Shengjun Xin = <sxin@gopivotal.= com> wrote:
add '-Dmapreduce.user.c= lasspath.first=3Dtrue' to your command and try again



On Wed, Apr 9, 2014 at 6:27 AM, Kim= Chew <kchew534@gmail.com> wrote:
It seems to me that in Had= oop 2.2.1, using the "libjars" option does not search the jars lo= cated in the the local file system but HDFS. For example,

hadoop jar target/myJar.jar Foo -libjars /home/kchew/test-libs/testJar.= jar /user/kchew/inputs/raw.vector /user/kchew/outputs hdfs://remoteNN:80= 20 remoteJT:8021

14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with pr= ocessName=3DJobTracker, sessionId=3D
14/04/08 15:11:02 INFO mapreduce.Jo= bSubmitter: Cleaning up the staging area file:/tmp/hadoop-kchew/mapred/stag= ing/kchew202924688/.staging/job_local202924688_0001
14/04/08 15:11:02 ERROR security.UserGroupInformation: PriviledgedActionExc= eption as:kchew (auth:SIMPLE) cause:java.io.FileNotFoundException: File doe= s not exist: hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar java.io.FileNotFoundException: File does not exist: hdfs:/remoteNN:8020/hom= e/kchew/test-libs/testJar.jar
=A0=A0=A0 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(Distrib= utedFileSystem.java:1110)
=A0=A0=A0 at org.apache.hadoop.hdfs.Distribute= dFileSystem$17.doCall(DistributedFileSystem.java:1102)
=A0=A0=A0 at org.= apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java= :81)
=A0=A0=A0 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(Dis= tributedFileSystem.java:1102)
=A0=A0=A0 at org.apache.hadoop.mapreduce.f= ilecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCache= Manager.java:288)
=A0=A0=A0 at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheMa= nager.getFileStatus(ClientDistributedCacheManager.java:224)
=A0=A0=A0 at= org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determ= ineTimestamps(ClientDistributedCacheManager.java:93)
=A0=A0=A0 at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheMa= nager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager= .java:57)
=A0=A0=A0 at org.apache.hadoop.mapreduce.JobSubmitter.copyAndC= onfigureFiles(JobSubmitter.java:264)

So under Hadoop 2.2.1, do I have to explicitly set some configura= tions so when using the "libjars" option it will copy the file to= hdfs from local fs?

TIA

Kim
=



--
Regards
Shengjun




CONFIDENTIALITY NOTICE
NOTICE: This= message is intended for the use of the individual or entity to which it is= addressed and may contain information that is confidential, privileged and= exempt from disclosure under applicable law. If the reader of this message= is not the intended recipient, you are hereby notified that any printing, = copying, dissemination, distribution, disclosure or forwarding of this comm= unication is strictly prohibited. If you have received this communication i= n error, please contact the sender immediately and delete it from your syst= em. Thank You.



CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --089e0149c04e1dc63604f72ecbb0--