Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EB85B9857 for ; Thu, 29 Nov 2012 15:46:45 +0000 (UTC) Received: (qmail 16284 invoked by uid 500); 29 Nov 2012 15:46:40 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 16213 invoked by uid 500); 29 Nov 2012 15:46:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 16202 invoked by uid 99); 29 Nov 2012 15:46:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Nov 2012 15:46:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of balijamahesh.mca@gmail.com designates 209.85.216.48 as permitted sender) Received: from [209.85.216.48] (HELO mail-qa0-f48.google.com) (209.85.216.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Nov 2012 15:46:31 +0000 Received: by mail-qa0-f48.google.com with SMTP id o19so489513qap.14 for ; Thu, 29 Nov 2012 07:46:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=2L/HN8brdIVn9uc4RgEQwWw/+x3pxwJsiRbOjeJ4LFs=; b=hf0rECCvePJdpgaurQoSvZ+gqv8nJnmK/uT6D70hjqGovZas/jY/I4FpxBshSErSCx 2DUG6sw47/CKlFdX2+V8OEGX2olhjmwBeKel0ybYdL4ny1wBD4EJ2ZIpGa60VoL7fdYE CI/iaDvYc/PyqvxKae3XvgkV/90z+p3LgdI07egrsUIQLOMUd8aLnD3RHQzXAcvmh0W0 gQBlUi0efjt/H+a2seAYh79sFAErz0uHS8pD+Q4etUrDz+N0y2nPsIfI5CB61Xnl9ER4 YV0Fdpp/jgu2JJAZCZTRHiYiNneUSlWLm2ft7Qz0w7xCquvKaAK/27FLb8V5IhUDJh2J IfZA== MIME-Version: 1.0 Received: by 10.229.205.72 with SMTP id fp8mr6185179qcb.7.1354203970995; Thu, 29 Nov 2012 07:46:10 -0800 (PST) Received: by 10.49.60.69 with HTTP; Thu, 29 Nov 2012 07:46:10 -0800 (PST) In-Reply-To: References: Date: Thu, 29 Nov 2012 21:16:10 +0530 Message-ID: Subject: Re: Trouble with Word Count example From: Mahesh Balija To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016e68dd9f8a507d904cfa42e9b X-Virus-Checked: Checked by ClamAV on apache.org --0016e68dd9f8a507d904cfa42e9b Content-Type: text/plain; charset=ISO-8859-1 Hi Sandeep, For me everything seems to be alright. Can you tell us how are you running this job? Best, Mahesh.B. Calsoft Labs. On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra wrote: > Hello everyone, > > Like most others I am also running into some problems while running my > word count example. > I tried the various suggestion available on internet, but I guess it;s > time to go on email :) > > Here is the error that I am getting: > 12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User > classes may not be found. See JobConf(Class) or JobConf#setJar(String). > 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to process > : 1 > 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop > library > 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is > available > 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded > 12/11/29 10:21:00 INFO mapred.JobClient: Running job: job_201210310210_0040 > 12/11/29 10:21:01 INFO mapred.JobClient: map 0% reduce 0% > 12/11/29 10:21:07 INFO mapred.JobClient: Task Id : > attempt_201210310210_0040_m_000000_0, Status : FAILED > java.lang.RuntimeException: *java.lang.ClassNotFoundException: Class > WordCount2$Map *not found > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439) > at > org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437) > ... 8 more > > And here is the source code: > > > import org.apache.hadoop.conf.Configuration; > import org.apache.hadoop.conf.Configured; > import org.apache.hadoop.fs.Path; > import org.apache.hadoop.io.IntWritable; > import org.apache.hadoop.io.LongWritable; > import org.apache.hadoop.io.Text; > import org.apache.hadoop.mapreduce.Job; > import org.apache.hadoop.mapreduce.Mapper; > import org.apache.hadoop.mapreduce.Reducer; > import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; > import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; > import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; > import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; > import org.apache.hadoop.util.Tool; > import org.apache.hadoop.util.ToolRunner; > > import java.io.IOException; > import java.util.StringTokenizer; > > public class WordCount2 extends Configured implements Tool { > > > public static class Map extends Mapper IntWritable> { > private final static IntWritable one = new IntWritable(1); > private Text word = new Text(); > > @Override > protected void map(LongWritable key, Text value, Context context) > throws IOException, InterruptedException { > String line = value.toString(); > StringTokenizer tokenizer = new StringTokenizer(line); > while (tokenizer.hasMoreTokens()) { > word.set(tokenizer.nextToken()); > context.write(word, one); > } > } > > } > > > public static class Reduce extends Reducer IntWritable> { > > @Override > protected void reduce(Text key, Iterable values, > Context context) throws IOException, InterruptedException { > > int sum = 0; > > for(IntWritable value : values) { > sum += value.get(); > } > // while (values.hasNext()) { > // sum += values.next().get(); > // } > context.write(key, new IntWritable(sum)); > } > > } > > @Override > public int run(String[] args) throws Exception { > Configuration conf = getConf(); > for (java.util.Map.Entry entry: conf) { > System.out.printf("%s=%s\n", entry.getKey(), entry.getValue()); > } > > System.out.println("arg[0]= "+args[0] + " args[1]= "+ args[1]); > > Job job = new Job(conf, WordCount2.class.getSimpleName()); > job.setJobName("wordcount2"); > job.setJarByClass(WordCount2.class); > > job.setMapOutputKeyClass(Text.class); > job.setMapOutputValueClass(IntWritable.class); > > job.setOutputKeyClass(Text.class); > job.setOutputValueClass(IntWritable.class); > > job.setMapperClass(Map.class); > job.setCombinerClass(Reduce.class); > job.setReducerClass(Reduce.class); > > job.setInputFormatClass(TextInputFormat.class); > job.setOutputFormatClass(TextOutputFormat.class); > > FileInputFormat.setInputPaths(job, new Path(args[0])); > FileOutputFormat.setOutputPath(job, new Path(args[1])); > > System.exit(job.waitForCompletion(true) ? 0 : 1); > > return 0; > } > > > > public static void main(String[] args) throws Exception { > int exitCode = ToolRunner.run(new WordCount2(), args); > System.exit(exitCode); > } > } > > > --0016e68dd9f8a507d904cfa42e9b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Sandeep,
=A0
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0For me everything seems to be alright= .
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Can you tell us how are you running thi= s job?
=A0
Best,
Mahesh.B.
Calsoft Labs.

On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra = <sandeepjangra@gmail.com> wrote:
Hello everyone,=20

=A0 Like most others I am also running into some problems while runnin= g my word count example.
=A0 I tried the various suggestion available on internet, but I guess = it;s time to go on email :)
=A0=A0
=A0 Here is the error that I am getting:
=A0=A012/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParse= r for parsing the arguments. Applications should implement Tool for the sam= e.
12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. =A0User = classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to pro= cess : 1
12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop= library
12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is ava= ilable
12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded=
12/11/29 10:21:00 INFO mapred.JobClient: Running job: job_201210310210= _0040
12/11/29 10:21:01 INFO mapred.JobClient: =A0map 0% reduce 0%
12/11/29 10:21:07 INFO mapred.JobClient: Task Id : attempt_20121031021= 0_0040_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class= WordCount2$Map not found
at org.apache.hadoop.conf.= Configuration.getClass(Configuration.java:1439)
at org.apache.hadoop.mapre= duce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
at org.apache.hadoop.mapre= d.MapTask.runNewMapper(MapTask.java:605)
at org.apache.hadoop.mapre= d.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapre= d.Child$4.run(Child.java:270)
at java.security.AccessCon= troller.doPrivileged(Native Method)
at javax.security.auth.Sub= ject.doAs(Subject.java:416)
at org.apache.hadoop.secur= ity.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapre= d.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not = found
at org.apache.hadoop.conf.= Configuration.getClassByName(Configuration.java:1350)
at org.apache.hadoop.conf.= Configuration.getClass(Configuration.java:1437)
... 8 more

And here is the source code:


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

import java.io.IOException;
import java.util.StringTokenizer;

public class WordCount2 extends Configured implements Tool {


=A0 =A0 public static class Map extends Mapper<LongWritable, Text, = Text, IntWritable> {
=A0 =A0 =A0 =A0 private final static IntWritable one =3D new IntWritab= le(1);
=A0 =A0 =A0 =A0 private Text word =3D new Text();

=A0 =A0 =A0 =A0 @Override
=A0 =A0 =A0 =A0 protected void map(LongWritable key, Text value, Conte= xt context) throws IOException, InterruptedException {
=A0 =A0 =A0 =A0 =A0 =A0 String line =3D value.toString();
=A0 =A0 =A0 =A0 =A0 =A0 StringTokenizer tokenizer =3D new StringTokeni= zer(line);
=A0 =A0 =A0 =A0 =A0 =A0 while (tokenizer.hasMoreTokens()) {
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 word.set(tokenizer.nextToken());
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 context.write(word, one);
=A0 =A0 =A0 =A0 =A0 =A0 }
=A0 =A0 =A0 =A0 }

=A0 =A0 }


=A0 =A0 public static class Reduce extends Reducer<Text, IntWritabl= e, Text, IntWritable> {

=A0 =A0 =A0 =A0 @Override
=A0 =A0 =A0 =A0 protected void reduce(Text key, Iterable<IntWritabl= e> values, Context context) throws IOException, InterruptedException {

=A0 =A0 =A0 =A0 =A0 =A0 int sum =3D 0;

=A0 =A0 =A0 =A0 =A0 =A0 for(IntWritable value : values) {
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sum +=3D value.get();
=A0 =A0 =A0 =A0 =A0 =A0 }
// =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0while (values.hasNext()) {
// =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sum +=3D values.next().get();
// =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A0 =A0 =A0 =A0 =A0 =A0 context.write(key, new IntWritable(sum));
=A0 =A0 =A0 =A0 }

=A0 =A0 }

=A0 =A0 @Override
=A0 =A0 public int run(String[] args) throws Exception {
=A0 =A0 =A0 =A0 Configuration conf =3D getConf();
=A0 =A0 =A0 =A0 for (java.util.Map.Entry<String, String> entry: = conf) {
=A0 =A0 =A0 =A0 =A0 =A0 System.out.printf("%s=3D%s\n", entry= .getKey(), entry.getValue());
=A0 =A0 =A0 =A0 }

=A0 =A0 =A0 =A0 System.out.println("arg[0]=3D "+args[0] + &q= uot; args[1]=3D "+ args[1]);

=A0 =A0 =A0 =A0 Job job =3D new Job(conf, WordCount2.class.getSimpleNa= me());
=A0 =A0 =A0 =A0 job.setJobName("wordcount2");
=A0 =A0 =A0 =A0 job.setJarByClass(WordCount2.class);

=A0 =A0 =A0 =A0 job.setMapOutputKeyClass(Text.class);
=A0 =A0 =A0 =A0 job.setMapOutputValueClass(IntWritable.class);

=A0 =A0 =A0 =A0 job.setOutputKeyClass(Text.class);
=A0 =A0 =A0 =A0 job.setOutputValueClass(IntWritable.class);

=A0 =A0 =A0 =A0 job.setMapperClass(Map.class);
=A0 =A0 =A0 =A0 job.setCombinerClass(Reduce.class);
=A0 =A0 =A0 =A0 job.setReducerClass(Reduce.class);

=A0 =A0 =A0 =A0 job.setInputFormatClass(TextInputFormat.class);
=A0 =A0 =A0 =A0 job.setOutputFormatClass(TextOutputFormat.class);

=A0 =A0 =A0 =A0 FileInputFormat.setInputPaths(job, new Path(args[0]));=
=A0 =A0 =A0 =A0 FileOutputFormat.setOutputPath(job, new Path(args[1]))= ;

=A0 =A0 =A0 =A0 System.exit(job.waitForCompletion(true) ? 0 : 1);

=A0 =A0 =A0 =A0 return 0;
=A0 =A0 }



=A0 =A0 public static void main(String[] args) throws Exception {
=A0 =A0 =A0 =A0 int exitCode =3D ToolRunner.run(new WordCount2(), args= );
=A0 =A0 =A0 =A0 System.exit(exitCode);
=A0 =A0 }
}



--0016e68dd9f8a507d904cfa42e9b--