hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ch huang <justlo...@gmail.com>
Subject Re: issue about no class find in running MR job
Date Fri, 13 Dec 2013 04:07:33 GMT
i find the reason
no main class found

# hadoop classpath
/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/usr/lib/hadoop/lib

normal

/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*::/usr/lib/hadoop/lib:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*

there exist ::   :) i just do not know the mean

On Fri, Dec 13, 2013 at 10:40 AM, ch huang <justlooks@gmail.com> wrote:

> no ,it need not , hadoop can run class directly,i try in other box,it work
> fine,
>
>
> # hadoop com/test/demo/WordCount
> Error: Could not find or load main class com.test.demo.WordCount
> [root@CHBM224 test]# hadoop classpath
>
> /etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*
> [root@CHBM224 test]# echo $CLASSPATH
> .:/usr/java/jdk1.7.0_25/lib/dt.jar:/usr/java/jdk1.7.0_25/lib/tools.jar
> ---copy class path to another box ,and it's work fine
>
> # cd test/
> [root@CH22 test]# hadoop com/test/demo/WordCount
>
> teragen <num rows> <output dir>
> [root@CH22 test]# hadoop classpath
>
> /etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*::/usr/lib/hadoop/lib:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*
> [root@CH22 test]# echo $CLASSPATH
> .:/usr/java/jdk1.6.0_35/lib/dt.jar:/usr/java/jdk1.6.0_35/lib/tools.jar
>
>
>  On Fri, Dec 13, 2013 at 10:17 AM, Tao Xiao <xiaotao.cs.nju@gmail.com>wrote:
>
>> how did you package and compile your jar ? did you specify the main class
>> for the JAR file you generated ?
>>
>>
>> 2013/12/13 ch huang <justlooks@gmail.com>
>>
>>> hi,maillist:
>>>
>>>              i rewrite WordCount.java and try to compile and run  it but
>>> it say not find main class ,why?
>>>
>>> [root@CHBM224 myMR]# cat WordCount.java
>>> import java.io.IOException;
>>> import java.util.StringTokenizer;
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>> import org.apache.hadoop.util.ToolRunner;
>>> import org.apache.hadoop.util.Tool;
>>> public class WordCount extends Configured implements Tool {
>>>   public static class TokenizerMapper
>>>        extends Mapper<Object, Text, Text, IntWritable>{
>>>     private final static IntWritable one = new IntWritable(1);
>>>     private Text word = new Text();
>>>     public void map(Object key, Text value, Context context
>>>                     ) throws IOException, InterruptedException {
>>>       StringTokenizer itr = new StringTokenizer(value.toString());
>>>       while (itr.hasMoreTokens()) {
>>>         word.set(itr.nextToken());
>>>         context.write(word, one);
>>>       }
>>>     }
>>>   }
>>>   public static class IntSumReducer
>>>        extends Reducer<Text,IntWritable,Text,IntWritable> {
>>>     private IntWritable result = new IntWritable();
>>>     public void reduce(Text key, Iterable<IntWritable> values,
>>>                        Context context
>>>                        ) throws IOException, InterruptedException {
>>>       int sum = 0;
>>>       for (IntWritable val : values) {
>>>         sum += val.get();
>>>       }
>>>       result.set(sum);
>>>       context.write(key, result);
>>>     }
>>>   }
>>>   private static void usage() throws IOException {
>>>     System.err.println("teragen <num rows> <output dir>");
>>>   }
>>>   public int run(String[] args) throws IOException,
>>> InterruptedException,
>>> ClassN
>>> otFoundException {
>>>         Job job = Job.getInstance(getConf());
>>>         if (args.length != 2) {
>>>               usage();
>>>               return 2;
>>>         }
>>>         job.setJobName("wordcount");
>>>         job.setJarByClass(WordCount.class);
>>>         job.setMapperClass(TokenizerMapper.class);
>>>         job.setCombinerClass(IntSumReducer.class);
>>>         job.setReducerClass(IntSumReducer.class);
>>>         job.setOutputKeyClass(Text.class);
>>>         job.setOutputValueClass(IntWritable.class);
>>>         FileInputFormat.addInputPath(job, new Path(args[0]));
>>>         FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>         return job.waitForCompletion(true) ? 0 : 1;
>>>   }
>>>   public static void main(String[] args) throws Exception {
>>>         int res = ToolRunner.run(new Configuration(), new WordCount(),
>>> args);
>>>         System.exit(res);
>>>   }
>>> }
>>> [root@CHBM224 myMR]# javac -cp
>>> '/usr/lib/hadoop/*:/usr/lib/hadoop-mapreduce/*' -d ../test WordCount.java
>>> [root@CHBM224 myMR]#  hadoop ../test/WordCount
>>> Error: Could not find or load main class ...test.WordCount
>>>
>>
>>
>

Mime
View raw message