Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7AB5EDE9C for ; Thu, 29 Nov 2012 19:14:33 +0000 (UTC) Received: (qmail 50834 invoked by uid 500); 29 Nov 2012 19:14:28 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 50744 invoked by uid 500); 29 Nov 2012 19:14:28 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 50737 invoked by uid 99); 29 Nov 2012 19:14:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Nov 2012 19:14:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [206.47.135.205] (HELO Spam1.prd.mpac.ca) (206.47.135.205) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Nov 2012 19:14:19 +0000 Received: from Spam1.prd.mpac.ca (unknown [127.0.0.1]) by IMSVA80 (Postfix) with ESMTP id A75C61D805B for ; Thu, 29 Nov 2012 14:13:55 -0500 (EST) Received: from SMAIL1.prd.mpac.ca (unknown [172.29.2.53]) by Spam1.prd.mpac.ca (Postfix) with ESMTP id 3CF8D1D8054 for ; Thu, 29 Nov 2012 14:13:54 -0500 (EST) Received: from SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb]) by SMAIL1.prd.mpac.ca ([fe80::18cb:8648:b77f:2b55%11]) with mapi id 14.02.0318.004; Thu, 29 Nov 2012 14:13:53 -0500 From: "Kartashov, Andy" To: "user@hadoop.apache.org" Subject: RE: Trouble with Word Count example Thread-Topic: Trouble with Word Count example Thread-Index: AQHNzkab8rFCdcGBjEOHHyOcBVr8yZgBSJIAgAACUYCAAC0xAP//tguw Date: Thu, 29 Nov 2012 19:13:52 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.29.60.102] Content-Type: multipart/alternative; boundary="_000_BD42F346AE90F544A731516A805D1B8AD8847DSMAIL1prdmpacca_" MIME-Version: 1.0 X-TM-AS-Product-Ver: IMSVA-8.0.0.1304-6.5.0.1024-19402.007 X-TM-AS-Result: No--29.677-5.0-31-10 X-imss-scan-details: No--29.677-5.0-31-10 X-TM-AS-Result-Xfilter: Match text exemption rules:No X-TMASE-MatchedRID: DuKherWvI/uI0KPyMNrNUuKXavbHY/C1tzhXKc4i3KuHlDUCu0I+XD/z IkvUoSsdhANTY9WAntGxYbXkY9OeyfUVVWSA4KqTlVHM/F6YkvTRahuPwaQ1WuVs8DNq02hCevu GN4aJZuWfGEe5X5RT0Bj83EN3CamEcdV1fv9AzTAbmaDSnOqZfnMeZN8H36Iz2viB/Jr4D1RBRV noG6NyjoQl5B0GOlIjwSfjByLqt7EMeUy96gk4BLMjW/sniEQKWuFm7TqL4eEqNag3uuW+4sflk +H44jBn8IzlwrY83Vm3usBucb8rBkQVckjz5U5evnSVh24OCE4k80hXoYXyaxtkYykBey0OaUXs 6FguVy3IpZGFijAxFbrYM2lQ0HNzAE+5eBl91IO4jAucHcCqndUEOicf335WLcVjbs+keaxdUKa OaTcrDPa+OY9C8+cOwSvAKf/0F8wTv1AH/7pVaTiU1nXLlwv+OhJ9m53n4aDku3RXZ7MJ7HP0pw QeaOOJjeYWuYc7ufeD3px9XNgl6jblc6Gei4nlG66tuBin4dCCS0QTA8JzpEn+ggZfLJfyqAVfb HPk2MfiPiWvhafhGKwQUp5/xlTG2ygMnPk4DNxjiC4p+/AIFjMffNZYCk3Dwb2pkFEdwy+GoSfE HPHpLHb4Bm7FqQnLXHn0K2CbIexqD9kNPFM9GJcJk+S0k5+DLZWWNVgH0Y03Rx11yoeUQt0pRIF bsjfbb7mJ6HbhlhHHkD3MU9pI3ZLK6Sl8R3+IbofUq4SMbd8K8tLLUlw11y7dpbMZwiCd6evgFm OgB7DPZ/cG93XSYw== X-Virus-Checked: Checked by ClamAV on apache.org --_000_BD42F346AE90F544A731516A805D1B8AD8847DSMAIL1prdmpacca_ Content-Type: text/plain; charset="us-ascii" Can you try running jar -tvf word_cnt.jar and see if your static nested classes WordCount2$Map.class and WordCount2$Reduce.class have actually been added to the jar. Rgds, AK47 From: Sandeep Jangra [mailto:sandeepjangra@gmail.com] Sent: Thursday, November 29, 2012 1:36 PM To: user@hadoop.apache.org Subject: Re: Trouble with Word Count example Also, I did set the HADOOP_CLASSPATH variable to point to the word_cnt.jar only. On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra > wrote: Thanks for the quick response Mahesh. I am using the following command: sudo -u hdfs hadoop jar word_cnt.jar WordCount2 /tmp/root/input /tmp/root/output15 -libjars=word_cnt.jar (The input directory exists on the hdfs) This is how I compiled and packaged it: javac -classpath /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/* -d word_cnt WordCount2.java jar -cvf word_cnt.jar -C word_cnt/ . On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija > wrote: Hi Sandeep, For me everything seems to be alright. Can you tell us how are you running this job? Best, Mahesh.B. Calsoft Labs. On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra > wrote: Hello everyone, Like most others I am also running into some problems while running my word count example. I tried the various suggestion available on internet, but I guess it;s time to go on email :) Here is the error that I am getting: 12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to process : 1 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is available 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded 12/11/29 10:21:00 INFO mapred.JobClient: Running job: job_201210310210_0040 12/11/29 10:21:01 INFO mapred.JobClient: map 0% reduce 0% 12/11/29 10:21:07 INFO mapred.JobClient: Task Id : attempt_201210310210_0040_m_000000_0, Status : FAILED java.lang.RuntimeException: java.lang.ClassNotFoundException: Class WordCount2$Map not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437) ... 8 more And here is the source code: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import java.io.IOException; import java.util.StringTokenizer; public class WordCount2 extends Configured implements Tool { public static class Map extends Mapper { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } public static class Reduce extends Reducer { @Override protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable value : values) { sum += value.get(); } // while (values.hasNext()) { // sum += values.next().get(); // } context.write(key, new IntWritable(sum)); } } @Override public int run(String[] args) throws Exception { Configuration conf = getConf(); for (java.util.Map.Entry entry: conf) { System.out.printf("%s=%s\n", entry.getKey(), entry.getValue()); } System.out.println("arg[0]= "+args[0] + " args[1]= "+ args[1]); Job job = new Job(conf, WordCount2.class.getSimpleName()); job.setJobName("wordcount2"); job.setJarByClass(WordCount2.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setCombinerClass(Reduce.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); return 0; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new WordCount2(), args); System.exit(exitCode); } } NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD8847DSMAIL1prdmpacca_ Content-Type: text/html; charset="us-ascii"

Can you try running jar –tvf word_cnt.jar and see if your static nested classes WordCount2$Map.class and WordCount2$Reduce.class have actually been added to the jar.

 

Rgds,

AK47

 

 

From: Sandeep Jangra [mailto:sandeepjangra@gmail.com]
Sent: Thursday, November 29, 2012 1:36 PM
To: user@hadoop.apache.org
Subject: Re: Trouble with Word Count example

 

Also, I did set the HADOOP_CLASSPATH variable to point to the word_cnt.jar only.

 

On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra <sandeepjangra@gmail.com> wrote:

Thanks for the quick response Mahesh.

 

I am using the following command:

 

sudo -u hdfs hadoop jar word_cnt.jar WordCount2  /tmp/root/input /tmp/root/output15  -libjars=word_cnt.jar

(The input directory exists on the hdfs)

 

This is how I compiled and packaged it:

 

javac -classpath /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/*  -d word_cnt WordCount2.java

jar -cvf word_cnt.jar -C word_cnt/ .

 

 

 

On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija <balijamahesh.mca@gmail.com> wrote:

Hi Sandeep,

 

           For me everything seems to be alright.

           Can you tell us how are you running this job?

 

Best,

Mahesh.B.

Calsoft Labs.

On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra <sandeepjangra@gmail.com> wrote:

Hello everyone,

 

  Like most others I am also running into some problems while running my word count example.

  I tried the various suggestion available on internet, but I guess it;s time to go on email :)

  

  Here is the error that I am getting:

  12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to process : 1

12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library

12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is available

12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded

12/11/29 10:21:00 INFO mapred.JobClient: Running job: job_201210310210_0040

12/11/29 10:21:01 INFO mapred.JobClient:  map 0% reduce 0%

12/11/29 10:21:07 INFO mapred.JobClient: Task Id : attempt_201210310210_0040_m_000000_0, Status : FAILED

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class WordCount2$Map not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439)

at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:416)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)

at org.apache.hadoop.mapred.Child.main(Child.java:264)

Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437)

... 8 more

 

And here is the source code:

 

 

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.conf.Configured;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import org.apache.hadoop.util.Tool;

import org.apache.hadoop.util.ToolRunner;

 

import java.io.IOException;

import java.util.StringTokenizer;

 

public class WordCount2 extends Configured implements Tool {

 

 

    public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);

        private Text word = new Text();

 

        @Override

        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

            String line = value.toString();

            StringTokenizer tokenizer = new StringTokenizer(line);

            while (tokenizer.hasMoreTokens()) {

                word.set(tokenizer.nextToken());

                context.write(word, one);

            }

        }

 

    }

 

 

    public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

 

        @Override

        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {

 

            int sum = 0;

 

            for(IntWritable value : values) {

                sum += value.get();

            }

//                    while (values.hasNext()) {

//                          sum += values.next().get();

//                        }

            context.write(key, new IntWritable(sum));

        }

 

    }

 

    @Override

    public int run(String[] args) throws Exception {

        Configuration conf = getConf();

        for (java.util.Map.Entry<String, String> entry: conf) {

            System.out.printf("%s=%s\n", entry.getKey(), entry.getValue());

        }

 

        System.out.println("arg[0]= "+args[0] + " args[1]= "+ args[1]);

 

        Job job = new Job(conf, WordCount2.class.getSimpleName());

        job.setJobName("wordcount2");

        job.setJarByClass(WordCount2.class);

 

        job.setMapOutputKeyClass(Text.class);

        job.setMapOutputValueClass(IntWritable.class);

 

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

 

        job.setMapperClass(Map.class);

        job.setCombinerClass(Reduce.class);

        job.setReducerClass(Reduce.class);

 

        job.setInputFormatClass(TextInputFormat.class);

        job.setOutputFormatClass(TextOutputFormat.class);

 

        FileInputFormat.setInputPaths(job, new Path(args[0]));

        FileOutputFormat.setOutputPath(job, new Path(args[1]));

 

        System.exit(job.waitForCompletion(true) ? 0 : 1);

 

        return 0;

    }

 

 

 

    public static void main(String[] args) throws Exception {

        int exitCode = ToolRunner.run(new WordCount2(), args);

        System.exit(exitCode);

    }

}

 

 

 

 

 

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel --_000_BD42F346AE90F544A731516A805D1B8AD8847DSMAIL1prdmpacca_--