hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: Trouble with Word Count example
Date Thu, 29 Nov 2012 21:04:46 GMT
Also check permissions you  are doing Sudo 

Sent from my iPhone

On Nov 29, 2012, at 1:00 PM, "Kartashov, Andy" <Andy.Kartashov@mpac.ca> wrote:

> Maybe you stepped out of your working directory. “$ ls –l”  Do you see your .jar?
>  
> From: Sandeep Jangra [mailto:sandeepjangra@gmail.com] 
> Sent: Thursday, November 29, 2012 3:46 PM
> To: user@hadoop.apache.org
> Subject: Re: Trouble with Word Count example
>  
> Hi Harsh,
>  
>   I tried putting the generic option first, but it throws exception file not found.
>   The jar is in current directory. Then I tried giving absolute path of this jar, but
that also brought no luck.
>  
>   sudo -u hdfs hadoop jar word_cnt.jar WordCount2  -libjars=word_cnt.jar /tmp/root/input
/tmp/root/output17 
> Exception in thread "main" java.io.FileNotFoundException: File word_cnt.jar does not
exist.
>             at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:384)
>             at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:280)
>             at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:418)
>             at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:168)
>             at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:151)
>             at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
>  
>   Also, I have been deleting my jars and the class directory before each new try. So
even I am suspicious why do I see this:
> "12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User classes may not
be found. See JobConf(Class) or JobConf#setJar(String)."
>  
>   Could it be that my hadoop is running on old jar files (the one with package name "mapred"
(not mapreduce))
>   But my program is using new jars as well.
>  
>   I can try going back to old word count example on the apache site and using old jars.
>  
>   Any other pointers would be highly appreciated. Thanks
>  
>   
> 
> On Thu, Nov 29, 2012 at 2:42 PM, Harsh J <harsh@cloudera.com> wrote:
> I think you may have not recompiled your application properly.
> 
> Your runtime shows this:
> 
> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 
> Which should not appear, cause your code has this (which I suspect you
> may have added later, accidentally?):
> 
> job.setJarByClass(WordCount2.class);
> 
> So if you can try deleting the older jar and recompiling it, the
> problem would go away.
> 
> Also, when passing generic options such as -libjars, etc., they need
> to go first in order. I mean, it should always be [Classname] [Generic
> Options] [Application Options]. Otherwise, they may not get utilized
> properly.
> 
> On Fri, Nov 30, 2012 at 12:51 AM, Sandeep Jangra
> <sandeepjangra@gmail.com> wrote:
> > Yups I can see my class files there.
> >
> >
> > On Thu, Nov 29, 2012 at 2:13 PM, Kartashov, Andy <Andy.Kartashov@mpac.ca>
> > wrote:
> >>
> >> Can you try running jar –tvf word_cnt.jar and see if your static nested
> >> classes WordCount2$Map.class and WordCount2$Reduce.class have actually been
> >> added to the jar.
> >>
> >>
> >>
> >> Rgds,
> >>
> >> AK47
> >>
> >>
> >>
> >>
> >>
> >> From: Sandeep Jangra [mailto:sandeepjangra@gmail.com]
> >> Sent: Thursday, November 29, 2012 1:36 PM
> >> To: user@hadoop.apache.org
> >> Subject: Re: Trouble with Word Count example
> >>
> >>
> >>
> >> Also, I did set the HADOOP_CLASSPATH variable to point to the word_cnt.jar
> >> only.
> >>
> >>
> >>
> >> On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra <sandeepjangra@gmail.com>
> >> wrote:
> >>
> >> Thanks for the quick response Mahesh.
> >>
> >>
> >>
> >> I am using the following command:
> >>
> >>
> >>
> >> sudo -u hdfs hadoop jar word_cnt.jar WordCount2  /tmp/root/input
> >> /tmp/root/output15  -libjars=word_cnt.jar
> >>
> >> (The input directory exists on the hdfs)
> >>
> >>
> >>
> >> This is how I compiled and packaged it:
> >>
> >>
> >>
> >> javac -classpath
> >> /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/*  -d
> >> word_cnt WordCount2.java
> >>
> >> jar -cvf word_cnt.jar -C word_cnt/ .
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija
> >> <balijamahesh.mca@gmail.com> wrote:
> >>
> >> Hi Sandeep,
> >>
> >>
> >>
> >>            For me everything seems to be alright.
> >>
> >>            Can you tell us how are you running this job?
> >>
> >>
> >>
> >> Best,
> >>
> >> Mahesh.B.
> >>
> >> Calsoft Labs.
> >>
> >> On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra <sandeepjangra@gmail.com>
> >> wrote:
> >>
> >> Hello everyone,
> >>
> >>
> >>
> >>   Like most others I am also running into some problems while running my
> >> word count example.
> >>
> >>   I tried the various suggestion available on internet, but I guess it;s
> >> time to go on email :)
> >>
> >>
> >>
> >>   Here is the error that I am getting:
> >>
> >>   12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for
> >> parsing the arguments. Applications should implement Tool for the same.
> >>
> >> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User
> >> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> >>
> >> 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to process
> >> : 1
> >>
> >> 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
> >> library
> >>
> >> 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is
> >> available
> >>
> >> 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded
> >>
> >> 12/11/29 10:21:00 INFO mapred.JobClient: Running job:
> >> job_201210310210_0040
> >>
> >> 12/11/29 10:21:01 INFO mapred.JobClient:  map 0% reduce 0%
> >>
> >> 12/11/29 10:21:07 INFO mapred.JobClient: Task Id :
> >> attempt_201210310210_0040_m_000000_0, Status : FAILED
> >>
> >> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> >> WordCount2$Map not found
> >>
> >> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439)
> >>
> >> at
> >> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
> >>
> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
> >>
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> >>
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> >>
> >> at java.security.AccessController.doPrivileged(Native Method)
> >>
> >> at javax.security.auth.Subject.doAs(Subject.java:416)
> >>
> >> at
> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> >>
> >> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> >>
> >> Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not
> >> found
> >>
> >> at
> >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350)
> >>
> >> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437)
> >>
> >> ... 8 more
> >>
> >>
> >>
> >> And here is the source code:
> >>
> >>
> >>
> >>
> >>
> >> import org.apache.hadoop.conf.Configuration;
> >>
> >> import org.apache.hadoop.conf.Configured;
> >>
> >> import org.apache.hadoop.fs.Path;
> >>
> >> import org.apache.hadoop.io.IntWritable;
> >>
> >> import org.apache.hadoop.io.LongWritable;
> >>
> >> import org.apache.hadoop.io.Text;
> >>
> >> import org.apache.hadoop.mapreduce.Job;
> >>
> >> import org.apache.hadoop.mapreduce.Mapper;
> >>
> >> import org.apache.hadoop.mapreduce.Reducer;
> >>
> >> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> >>
> >> import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
> >>
> >> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> >>
> >> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
> >>
> >> import org.apache.hadoop.util.Tool;
> >>
> >> import org.apache.hadoop.util.ToolRunner;
> >>
> >>
> >>
> >> import java.io.IOException;
> >>
> >> import java.util.StringTokenizer;
> >>
> >>
> >>
> >> public class WordCount2 extends Configured implements Tool {
> >>
> >>
> >>
> >>
> >>
> >>     public static class Map extends Mapper<LongWritable, Text, Text,
> >> IntWritable> {
> >>
> >>         private final static IntWritable one = new IntWritable(1);
> >>
> >>         private Text word = new Text();
> >>
> >>
> >>
> >>         @Override
> >>
> >>         protected void map(LongWritable key, Text value, Context context)
> >> throws IOException, InterruptedException {
> >>
> >>             String line = value.toString();
> >>
> >>             StringTokenizer tokenizer = new StringTokenizer(line);
> >>
> >>             while (tokenizer.hasMoreTokens()) {
> >>
> >>                 word.set(tokenizer.nextToken());
> >>
> >>                 context.write(word, one);
> >>
> >>             }
> >>
> >>         }
> >>
> >>
> >>
> >>     }
> >>
> >>
> >>
> >>
> >>
> >>     public static class Reduce extends Reducer<Text, IntWritable, Text,
> >> IntWritable> {
> >>
> >>
> >>
> >>         @Override
> >>
> >>         protected void reduce(Text key, Iterable<IntWritable> values,
> >> Context context) throws IOException, InterruptedException {
> >>
> >>
> >>
> >>             int sum = 0;
> >>
> >>
> >>
> >> &n

Mime
View raw message