hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avinash Lakshman <alaksh...@facebook.com>
Subject Re: Classpath setting problem for Map tasks
Date Tue, 29 May 2007 22:02:50 GMT
Thanks a million. Everything works now.

Thanks
Avinash


On 5/29/07 1:51 PM, "Mahadev Konar" <mahadev@yahoo-inc.com> wrote:

> Hi Avinash,
>   Java org.apache.... will not work. This does not copy the job.jar and
> that's why you run into these Classnotfoundexcpetions.
> The way to run the wordocunt example is:
> Bin/hadoop jar hadoop-examples.jar wordcount .......
> 
> This way the jobclient is called by the hadoop command to transfer the
> job.jar and get the status of the job.
> 
> Regards
> Mahadev
> 
>> > -----Original Message-----
>> > From: Avinash Lakshman [mailto:alakshman@facebook.com]
>> > Sent: Tuesday, May 29, 2007 1:30 PM
>> > To: hadoop-user@lucene.apache.org
>> > Subject: Re: Classpath setting problem for Map tasks
>> >
>> > Hi Mahadev
>> >
>> > This is how I am running it. I created a test data file and put it in HDFS
>> > using ./hadoop fs -put command. Then I modified my WordCount sample to add
>> > two lines to the main which look like this :
>> >
>> > conf.set("fs.default.name", "hdfs://dev030.sctm.facebook.com:9000");
>> > conf.set("mapred.job.tracker", "dev030.sctm.facebook.com:50029");
>> >
>> > Then I run the sample by executing the command :
>> >
>> > java org.apache.hadoop.examples.WordCount -m 10 -r 4
>> > /user/alakshman/test.dat ~/output-dir
>> >
>> > where the ~/output-dir is the local file system while the
>> > /user/alakshman/test.dat is the path in HDFS.
>> >
>> > This is when I get a whole slew of classnotfoundexceptions. Is the process
>> > incorrect ?
>> >
>> > Thanks
>> > Avinash
>> >
>> >
>> >
>> > On 5/29/07 1:17 PM, "Mahadev Konar" <mahadev@yahoo-inc.com> wrote:
>> >
>>> > > Hi Avinash,
>>> > >  Can you explain how you are running the wordcount example? Also, which
>>> > > version of Hadoop are you using? I would really recommend using 0.12.3.
>>> > >
>>> > > Regards
>>> > > Mahadev
>>> > >
>>>>> > >> > -----Original Message-----
>>>>> > >> > From: Phantom [mailto:ghostwhoowalks@gmail.com]
>>>>> > >> > Sent: Tuesday, May 29, 2007 1:10 PM
>>>>> > >> > To: hadoop-user@lucene.apache.org
>>>>> > >> > Subject: Classpath setting problem for Map tasks
>>>>> > >> >
>>>>> > >> > How do I specify the classpath via the JobConf ? In
my jobtracker
web
>> > UI I
>>>>> > >> > see a whole slew of ClassNotFoundException's for the
>> > WordCount$MapClass.
>>>>> > >> > The
>>>>> > >> > stack trace is pasted below. But how do I fix this
- just pass the
-
>> > cp
>>>>> > >> > <classpath> to the mapred.child.java.opts property
of the JobConf ?
>>>>> > >> > exporting CLASSPATH doesn't seem to work.hh
>>>>> > >> >
>>>>> > >> > Thanks
>>>>> > >> > Avinash
>>>>> > >> >
>>>>> > >> > java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>>>>> > >> > )
>>>>> > >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>>>>> > >> > Caused by: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>>>>> > >> >       ... 6 more
>>>>> > >> > Caused by: java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> > >> >       at java.security.AccessController.doPrivileged(Native
Method)
>>>>> > >> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> > >> >       at
>> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>> > >> >       at
>> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>>>> > >> >       at java.lang.Class.forName0(Native Method)
>>>>> > >> >       at java.lang.Class.forName(Class.java:242)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>>>>> > >> > )
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>>>>> > >> >       ... 7 more
>>>>> > >> >
>>>>> > >> > java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>>>>> > >> > )
>>>>> > >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>>>>> > >> > Caused by: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>>>>> > >> >       ... 6 more
>>>>> > >> > Caused by: java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> > >> >       at java.security.AccessController.doPrivileged(Native
Method)
>>>>> > >> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> > >> >       at
>> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>> > >> >       at
>> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>>>> > >> >       at java.lang.Class.forName0(Native Method)
>>>>> > >> >       at java.lang.Class.forName(Class.java:242)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>>>>> > >> > )
>>>>> > >> >       at
>>>>> > >> > 
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>>>>> > >> >       ... 7 more
>>> > >
>>> > >
>> >
> 
> 
> 



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message