hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Jangra <sandeepjan...@gmail.com>
Subject Re: Trouble with Word Count example
Date Fri, 30 Nov 2012 01:22:53 GMT
Yes, I am working on two versions old and new. if I have any luck, I will
for sure email my findings back.
Thanks for this input on libjars.



On Thu, Nov 29, 2012 at 7:55 PM, Mahesh Balija
<balijamahesh.mca@gmail.com>wrote:

> Hi Sandeep,
>
>             One important thing here is, if you are passing your jar with
> hadoop jar your.jar then it is absolutely NOT required to pass it once
> again with -libjars option.
>             As Harsh said do recompile once again and see running your
> command without using -libjars option.
>             Also you can try your luck by running the JOB in old and new
> versions.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
>
> On Fri, Nov 30, 2012 at 2:16 AM, Sandeep Jangra <sandeepjangra@gmail.com>wrote:
>
>> Hi Harsh,
>>
>>   I tried putting the generic option first, but it throws exception file
>> not found.
>>   The jar is in current directory. Then I tried giving absolute path of
>> this jar, but that also brought no luck.
>>
>>   sudo -u hdfs hadoop jar word_cnt.jar WordCount2  -libjars=word_cnt.jar
>> /tmp/root/input /tmp/root/output17
>> Exception in thread "main" java.io.FileNotFoundException: File
>> word_cnt.jar does not exist.
>>  at
>> org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:384)
>> at
>> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:280)
>>  at
>> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:418)
>> at
>> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:168)
>>  at
>> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:151)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
>>
>>   Also, I have been deleting my jars and the class directory before each
>> new try. So even I am suspicious why do I see this:
>> "12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.
>>  User classes may not be found. See JobConf(Class)
>> or JobConf#setJar(String)."
>>
>>   Could it be that my hadoop is running on old jar files (the one with
>> package name "mapred" (not mapreduce))
>>   But my program is using new jars as well.
>>
>>   I can try going back to old word count example on the apache site and
>> using old jars.
>>
>>   Any other pointers would be highly appreciated. Thanks
>>
>>
>>
>> On Thu, Nov 29, 2012 at 2:42 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>>> I think you may have not recompiled your application properly.
>>>
>>> Your runtime shows this:
>>>
>>> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User
>>> classes may not be found. See JobConf(Class) or
>>> JobConf#setJar(String).
>>>
>>> Which should not appear, cause your code has this (which I suspect you
>>> may have added later, accidentally?):
>>>
>>> job.setJarByClass(WordCount2.class);
>>>
>>> So if you can try deleting the older jar and recompiling it, the
>>> problem would go away.
>>>
>>> Also, when passing generic options such as -libjars, etc., they need
>>> to go first in order. I mean, it should always be [Classname] [Generic
>>> Options] [Application Options]. Otherwise, they may not get utilized
>>> properly.
>>>
>>> On Fri, Nov 30, 2012 at 12:51 AM, Sandeep Jangra
>>> <sandeepjangra@gmail.com> wrote:
>>> > Yups I can see my class files there.
>>> >
>>> >
>>> > On Thu, Nov 29, 2012 at 2:13 PM, Kartashov, Andy <
>>> Andy.Kartashov@mpac.ca>
>>> > wrote:
>>> >>
>>> >> Can you try running jar –tvf word_cnt.jar and see if your static
>>> nested
>>> >> classes WordCount2$Map.class and WordCount2$Reduce.class have
>>> actually been
>>> >> added to the jar.
>>> >>
>>> >>
>>> >>
>>> >> Rgds,
>>> >>
>>> >> AK47
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> From: Sandeep Jangra [mailto:sandeepjangra@gmail.com]
>>> >> Sent: Thursday, November 29, 2012 1:36 PM
>>> >> To: user@hadoop.apache.org
>>> >> Subject: Re: Trouble with Word Count example
>>> >>
>>> >>
>>> >>
>>> >> Also, I did set the HADOOP_CLASSPATH variable to point to the
>>> word_cnt.jar
>>> >> only.
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra <
>>> sandeepjangra@gmail.com>
>>> >> wrote:
>>> >>
>>> >> Thanks for the quick response Mahesh.
>>> >>
>>> >>
>>> >>
>>> >> I am using the following command:
>>> >>
>>> >>
>>> >>
>>> >> sudo -u hdfs hadoop jar word_cnt.jar WordCount2  /tmp/root/input
>>> >> /tmp/root/output15  -libjars=word_cnt.jar
>>> >>
>>> >> (The input directory exists on the hdfs)
>>> >>
>>> >>
>>> >>
>>> >> This is how I compiled and packaged it:
>>> >>
>>> >>
>>> >>
>>> >> javac -classpath
>>> >> /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/*  -d
>>> >> word_cnt WordCount2.java
>>> >>
>>> >> jar -cvf word_cnt.jar -C word_cnt/ .
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija
>>> >> <balijamahesh.mca@gmail.com> wrote:
>>> >>
>>> >> Hi Sandeep,
>>> >>
>>> >>
>>> >>
>>> >>            For me everything seems to be alright.
>>> >>
>>> >>            Can you tell us how are you running this job?
>>> >>
>>> >>
>>> >>
>>> >> Best,
>>> >>
>>> >> Mahesh.B.
>>> >>
>>> >> Calsoft Labs.
>>> >>
>>> >> On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra <
>>> sandeepjangra@gmail.com>
>>> >> wrote:
>>> >>
>>> >> Hello everyone,
>>> >>
>>> >>
>>> >>
>>> >>   Like most others I am also running into some problems while running
>>> my
>>> >> word count example.
>>> >>
>>> >>   I tried the various suggestion available on internet, but I guess
>>> it;s
>>> >> time to go on email :)
>>> >>
>>> >>
>>> >>
>>> >>   Here is the error that I am getting:
>>> >>
>>> >>   12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser
>>> for
>>> >> parsing the arguments. Applications should implement Tool for the
>>> same.
>>> >>
>>> >> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User
>>> >> classes may not be found. See JobConf(Class) or
>>> JobConf#setJar(String).
>>> >>
>>> >> 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to
>>> process
>>> >> : 1
>>> >>
>>> >> 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>> >> library
>>> >>
>>> >> 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is
>>> >> available
>>> >>
>>> >> 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded
>>> >>
>>> >> 12/11/29 10:21:00 INFO mapred.JobClient: Running job:
>>> >> job_201210310210_0040
>>> >>
>>> >> 12/11/29 10:21:01 INFO mapred.JobClient:  map 0% reduce 0%
>>> >>
>>> >> 12/11/29 10:21:07 INFO mapred.JobClient: Task Id :
>>> >> attempt_201210310210_0040_m_000000_0, Status : FAILED
>>> >>
>>> >> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>>> >> WordCount2$Map not found
>>> >>
>>> >> at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439)
>>> >>
>>> >> at
>>> >>
>>> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
>>> >>
>>> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
>>> >>
>>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>>> >>
>>> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>> >>
>>> >> at java.security.AccessController.doPrivileged(Native Method)
>>> >>
>>> >> at javax.security.auth.Subject.doAs(Subject.java:416)
>>> >>
>>> >> at
>>> >>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>> >>
>>> >> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>> >>
>>> >> Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not
>>> >> found
>>> >>
>>> >> at
>>> >>
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350)
>>> >>
>>> >> at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437)
>>> >>
>>> >> ... 8 more
>>> >>
>>> >>
>>> >>
>>> >> And here is the source code:
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> import org.apache.hadoop.conf.Configuration;
>>> >>
>>> >> import org.apache.hadoop.conf.Configured;
>>> >>
>>> >> import org.apache.hadoop.fs.Path;
>>> >>
>>> >> import org.apache.hadoop.io.IntWritable;
>>> >>
>>> >> import org.apache.hadoop.io.LongWritable;
>>> >>
>>> >> import org.apache.hadoop.io.Text;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.Job;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.Mapper;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.Reducer;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> >>
>>> >> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
>>> >>
>>> >> import org.apache.hadoop.util.Tool;
>>> >>
>>> >> import org.apache.hadoop.util.ToolRunner;
>>> >>
>>> >>
>>> >>
>>> >> import java.io.IOException;
>>> >>
>>> >> import java.util.StringTokenizer;
>>> >>
>>> >>
>>> >>
>>> >> public class WordCount2 extends Configured implements Tool {
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>     public static class Map extends Mapper<LongWritable, Text, Text,
>>> >> IntWritable> {
>>> >>
>>> >>         private final static IntWritable one = new IntWritable(1);
>>> >>
>>> >>         private Text word = new Text();
>>> >>
>>> >>
>>> >>
>>> >>         @Override
>>> >>
>>> >>         protected void map(LongWritable key, Text value, Context
>>> context)
>>> >> throws IOException, InterruptedException {
>>> >>
>>> >>             String line = value.toString();
>>> >>
>>> >>             StringTokenizer tokenizer = new StringTokenizer(line);
>>> >>
>>> >>             while (tokenizer.hasMoreTokens()) {
>>> >>
>>> >>                 word.set(tokenizer.nextToken());
>>> >>
>>> >>                 context.write(word, one);
>>> >>
>>> >>             }
>>> >>
>>> >>         }
>>> >>
>>> >>
>>> >>
>>> >>     }
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>     public static class Reduce extends Reducer<Text, IntWritable,
>>> Text,
>>> >> IntWritable> {
>>> >>
>>> >>
>>> >>
>>> >>         @Override
>>> >>
>>> >>         protected void reduce(Text key, Iterable<IntWritable>
values,
>>> >> Context context) throws IOException, InterruptedException {
>>> >>
>>> >>
>>> >>
>>> >>             int sum = 0;
>>> >>
>>> >>
>>> >>
>>> >>             for(IntWritable value : values) {
>>> >>
>>> >>                 sum += value.get();
>>> >>
>>> >>             }
>>> >>
>>> >> //                    while (values.hasNext()) {
>>> >>
>>> >> //                          sum += values.next().get();
>>> >>
>>> >> //                        }
>>> >>
>>> >>             context.write(key, new IntWritable(sum));
>>> >>
>>> >>         }
>>> >>
>>> >>
>>> >>
>>> >>     }
>>> >>
>>> >>
>>> >>
>>> >>     @Override
>>> >>
>>> >>     public int run(String[] args) throws Exception {
>>> >>
>>> >>         Configuration conf = getConf();
>>> >>
>>> >>         for (java.util.Map.Entry<String, String> entry: conf)
{
>>> >>
>>> >>             System.out.printf("%s=%s\n", entry.getKey(),
>>> >> entry.getValue());
>>> >>
>>> >>         }
>>> >>
>>> >>
>>> >>
>>> >>         System.out.println("arg[0]= "+args[0] + " args[1]= "+
>>> args[1]);
>>> >>
>>> >>
>>> >>
>>> >>         Job job = new Job(conf, WordCount2.class.getSimpleName());
>>> >>
>>> >>         job.setJobName("wordcount2");
>>> >>
>>> >>         job.setJarByClass(WordCount2.class);
>>> >>
>>> >>
>>> >>
>>> >>         job.setMapOutputKeyClass(Text.class);
>>> >>
>>> >>         job.setMapOutputValueClass(IntWritable.class);
>>> >>
>>> >>
>>> >>
>>> >>         job.setOutputKeyClass(Text.class);
>>> >>
>>> >>         job.setOutputValueClass(IntWritable.class);
>>> >>
>>> >>
>>> >>
>>> >>         job.setMapperClass(Map.class);
>>> >>
>>> >>         job.setCombinerClass(Reduce.class);
>>> >>
>>> >>         job.setReducerClass(Reduce.class);
>>> >>
>>> >>
>>> >>
>>> >>         job.setInputFormatClass(TextInputFormat.class);
>>> >>
>>> >>         job.setOutputFormatClass(TextOutputFormat.class);
>>> >>
>>> >>
>>> >>
>>> >>         FileInputFormat.setInputPaths(job, new Path(args[0]));
>>> >>
>>> >>         FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>> >>
>>> >>
>>> >>
>>> >>         System.exit(job.waitForCompletion(true) ? 0 : 1);
>>> >>
>>> >>
>>> >>
>>> >>         return 0;
>>> >>
>>> >>     }
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>     public static void main(String[] args) throws Exception {
>>> >>
>>> >>         int exitCode = ToolRunner.run(new WordCount2(), args);
>>> >>
>>> >>         System.exit(exitCode);
>>> >>
>>> >>     }
>>> >>
>>> >> }
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> NOTICE: This e-mail message and any attachments are confidential,
>>> subject
>>> >> to copyright and may be privileged. Any unauthorized use, copying or
>>> >> disclosure is prohibited. If you are not the intended recipient,
>>> please
>>> >> delete and contact the sender immediately. Please consider the
>>> environment
>>> >> before printing this e-mail. AVIS : le présent courriel et toute pièce
>>> >> jointe qui l'accompagne sont confidentiels, protégés par le droit
>>> d'auteur
>>> >> et peuvent être couverts par le secret professionnel. Toute
>>> utilisation,
>>> >> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas
>>> le
>>> >> destinataire prévu de ce courriel, supprimez-le et contactez
>>> immédiatement
>>> >> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le
>>> présent
>>> >> courriel
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>

Mime
View raw message