hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Trouble with Word Count example
Date Thu, 29 Nov 2012 19:42:15 GMT
I think you may have not recompiled your application properly.

Your runtime shows this:

12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User
classes may not be found. See JobConf(Class) or
JobConf#setJar(String).

Which should not appear, cause your code has this (which I suspect you
may have added later, accidentally?):

job.setJarByClass(WordCount2.class);

So if you can try deleting the older jar and recompiling it, the
problem would go away.

Also, when passing generic options such as -libjars, etc., they need
to go first in order. I mean, it should always be [Classname] [Generic
Options] [Application Options]. Otherwise, they may not get utilized
properly.

On Fri, Nov 30, 2012 at 12:51 AM, Sandeep Jangra
<sandeepjangra@gmail.com> wrote:
> Yups I can see my class files there.
>
>
> On Thu, Nov 29, 2012 at 2:13 PM, Kartashov, Andy <Andy.Kartashov@mpac.ca>
> wrote:
>>
>> Can you try running jar –tvf word_cnt.jar and see if your static nested
>> classes WordCount2$Map.class and WordCount2$Reduce.class have actually been
>> added to the jar.
>>
>>
>>
>> Rgds,
>>
>> AK47
>>
>>
>>
>>
>>
>> From: Sandeep Jangra [mailto:sandeepjangra@gmail.com]
>> Sent: Thursday, November 29, 2012 1:36 PM
>> To: user@hadoop.apache.org
>> Subject: Re: Trouble with Word Count example
>>
>>
>>
>> Also, I did set the HADOOP_CLASSPATH variable to point to the word_cnt.jar
>> only.
>>
>>
>>
>> On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra <sandeepjangra@gmail.com>
>> wrote:
>>
>> Thanks for the quick response Mahesh.
>>
>>
>>
>> I am using the following command:
>>
>>
>>
>> sudo -u hdfs hadoop jar word_cnt.jar WordCount2  /tmp/root/input
>> /tmp/root/output15  -libjars=word_cnt.jar
>>
>> (The input directory exists on the hdfs)
>>
>>
>>
>> This is how I compiled and packaged it:
>>
>>
>>
>> javac -classpath
>> /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/*  -d
>> word_cnt WordCount2.java
>>
>> jar -cvf word_cnt.jar -C word_cnt/ .
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija
>> <balijamahesh.mca@gmail.com> wrote:
>>
>> Hi Sandeep,
>>
>>
>>
>>            For me everything seems to be alright.
>>
>>            Can you tell us how are you running this job?
>>
>>
>>
>> Best,
>>
>> Mahesh.B.
>>
>> Calsoft Labs.
>>
>> On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra <sandeepjangra@gmail.com>
>> wrote:
>>
>> Hello everyone,
>>
>>
>>
>>   Like most others I am also running into some problems while running my
>> word count example.
>>
>>   I tried the various suggestion available on internet, but I guess it;s
>> time to go on email :)
>>
>>
>>
>>   Here is the error that I am getting:
>>
>>   12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>>
>> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User
>> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
>>
>> 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to process
>> : 1
>>
>> 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>>
>> 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is
>> available
>>
>> 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded
>>
>> 12/11/29 10:21:00 INFO mapred.JobClient: Running job:
>> job_201210310210_0040
>>
>> 12/11/29 10:21:01 INFO mapred.JobClient:  map 0% reduce 0%
>>
>> 12/11/29 10:21:07 INFO mapred.JobClient: Task Id :
>> attempt_201210310210_0040_m_000000_0, Status : FAILED
>>
>> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>> WordCount2$Map not found
>>
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439)
>>
>> at
>> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
>>
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
>>
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>>
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>
>> Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not
>> found
>>
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350)
>>
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437)
>>
>> ... 8 more
>>
>>
>>
>> And here is the source code:
>>
>>
>>
>>
>>
>> import org.apache.hadoop.conf.Configuration;
>>
>> import org.apache.hadoop.conf.Configured;
>>
>> import org.apache.hadoop.fs.Path;
>>
>> import org.apache.hadoop.io.IntWritable;
>>
>> import org.apache.hadoop.io.LongWritable;
>>
>> import org.apache.hadoop.io.Text;
>>
>> import org.apache.hadoop.mapreduce.Job;
>>
>> import org.apache.hadoop.mapreduce.Mapper;
>>
>> import org.apache.hadoop.mapreduce.Reducer;
>>
>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>
>> import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
>>
>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>
>> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
>>
>> import org.apache.hadoop.util.Tool;
>>
>> import org.apache.hadoop.util.ToolRunner;
>>
>>
>>
>> import java.io.IOException;
>>
>> import java.util.StringTokenizer;
>>
>>
>>
>> public class WordCount2 extends Configured implements Tool {
>>
>>
>>
>>
>>
>>     public static class Map extends Mapper<LongWritable, Text, Text,
>> IntWritable> {
>>
>>         private final static IntWritable one = new IntWritable(1);
>>
>>         private Text word = new Text();
>>
>>
>>
>>         @Override
>>
>>         protected void map(LongWritable key, Text value, Context context)
>> throws IOException, InterruptedException {
>>
>>             String line = value.toString();
>>
>>             StringTokenizer tokenizer = new StringTokenizer(line);
>>
>>             while (tokenizer.hasMoreTokens()) {
>>
>>                 word.set(tokenizer.nextToken());
>>
>>                 context.write(word, one);
>>
>>             }
>>
>>         }
>>
>>
>>
>>     }
>>
>>
>>
>>
>>
>>     public static class Reduce extends Reducer<Text, IntWritable, Text,
>> IntWritable> {
>>
>>
>>
>>         @Override
>>
>>         protected void reduce(Text key, Iterable<IntWritable> values,
>> Context context) throws IOException, InterruptedException {
>>
>>
>>
>>             int sum = 0;
>>
>>
>>
>>             for(IntWritable value : values) {
>>
>>                 sum += value.get();
>>
>>             }
>>
>> //                    while (values.hasNext()) {
>>
>> //                          sum += values.next().get();
>>
>> //                        }
>>
>>             context.write(key, new IntWritable(sum));
>>
>>         }
>>
>>
>>
>>     }
>>
>>
>>
>>     @Override
>>
>>     public int run(String[] args) throws Exception {
>>
>>         Configuration conf = getConf();
>>
>>         for (java.util.Map.Entry<String, String> entry: conf) {
>>
>>             System.out.printf("%s=%s\n", entry.getKey(),
>> entry.getValue());
>>
>>         }
>>
>>
>>
>>         System.out.println("arg[0]= "+args[0] + " args[1]= "+ args[1]);
>>
>>
>>
>>         Job job = new Job(conf, WordCount2.class.getSimpleName());
>>
>>         job.setJobName("wordcount2");
>>
>>         job.setJarByClass(WordCount2.class);
>>
>>
>>
>>         job.setMapOutputKeyClass(Text.class);
>>
>>         job.setMapOutputValueClass(IntWritable.class);
>>
>>
>>
>>         job.setOutputKeyClass(Text.class);
>>
>>         job.setOutputValueClass(IntWritable.class);
>>
>>
>>
>>         job.setMapperClass(Map.class);
>>
>>         job.setCombinerClass(Reduce.class);
>>
>>         job.setReducerClass(Reduce.class);
>>
>>
>>
>>         job.setInputFormatClass(TextInputFormat.class);
>>
>>         job.setOutputFormatClass(TextOutputFormat.class);
>>
>>
>>
>>         FileInputFormat.setInputPaths(job, new Path(args[0]));
>>
>>         FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>
>>
>>
>>         System.exit(job.waitForCompletion(true) ? 0 : 1);
>>
>>
>>
>>         return 0;
>>
>>     }
>>
>>
>>
>>
>>
>>
>>
>>     public static void main(String[] args) throws Exception {
>>
>>         int exitCode = ToolRunner.run(new WordCount2(), args);
>>
>>         System.exit(exitCode);
>>
>>     }
>>
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> NOTICE: This e-mail message and any attachments are confidential, subject
>> to copyright and may be privileged. Any unauthorized use, copying or
>> disclosure is prohibited. If you are not the intended recipient, please
>> delete and contact the sender immediately. Please consider the environment
>> before printing this e-mail. AVIS : le présent courriel et toute pièce
>> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
>> et peuvent être couverts par le secret professionnel. Toute utilisation,
>> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
>> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
>> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
>> courriel
>
>



-- 
Harsh J

Mime
View raw message