hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bing Li <lbl...@gmail.com>
Subject Re: IOException: Job Failed
Date Thu, 09 Feb 2012 15:54:59 GMT
Dear all,

I got the problem. The input file is set wrongly.

Sorry for sending such an email!

Thanks,
Bing

On Thu, Feb 9, 2012 at 10:05 PM, Bing Li <lblabs@gmail.com> wrote:

> Dear all,
>
> When running some sample codes from Hadoop in Action, I got an
> IOException: Job Failed.
>
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at com.greatfree.testing.hadoop.CitationHistogram.run(Unknown Source)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at com.greatfree.testing.hadoop.CitationHistogram.main(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> For the following code, everything is fine.
>
>         ......
>
>         public static class MapClass extends MapReduceBase implements
> Mapper<Text, Text, Text, Text>
>         {
>                 public void map(Text key, Text value,
> OutputCollector<Text, Text> output, Reporter reporter) throws IOException
>                 {
>                         output.collect(value, key);
>                 }
>         }
>
>         public static class ReduceClass extends MapReduceBase implements
> Reducer<Text, Text, Text, IntWritable>
>         {
>                 public void reduce(Text key, Iterator<Text> values,
> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> IOException
>                 {
>                         int count = 0;
>                         while (values.hasNext())
>                         {
>                                 values.next();
>                                 count ++;
>                         }
>                         output.collect(key, new IntWritable(count));
>                 }
>         }
>
>         public int run(String[] args) throws Exception
>         {
>                 Configuration conf = getConf();
>
>                 JobConf job = new JobConf(conf, MyJob.class);
>
>                 Path in = new Path(args[0]);
>                 Path out = new Path(args[1]);
>                 FileInputFormat.setInputPaths(job, in);
>                 FileOutputFormat.setOutputPath(job, out);
>
>                 job.setJobName("MyJobForCount");
>                 job.setMapperClass(MapClass.class);
>                 job.setReducerClass(ReduceClass.class);
>
>                 job.setInputFormat(KeyValueTextInputFormat.class);
>                 job.setOutputFormat(TextOutputFormat.class);
>                 job.setOutputKeyClass(Text.class);
>                 job.setOutputValueClass(Text.class);
>                 job.set("key.value.separator.in.input.line", ",");
>
>                 JobClient.runJob(job);
>                 return 0;
>         }
>
>         ......
>
> However, when making some changes on the above code as follows, I got the
> above exception.
>
>         ......
>
>         public static class MapClass extends MapReduceBase implements
> Mapper<Text, Text, IntWritable, IntWritable>
>         {
>             private final static IntWritable uno = new IntWritable(1);
>             private IntWritable citationCount = new IntWritable();
>
>             public void map(Text key, Text value,
> OutputCollector<IntWritable, IntWritable> output, Reporter reporter) throws
> IOException
>             {
>                 citationCount.set(Integer.parseInt(value.toString()));
>                 output.collect(citationCount, uno);
>             }
>         }
>
>        public static class Reduce extends MapReduceBase implements
> Reducer<IntWritable,IntWritable,IntWritable,IntWritable>
>        {
>             public void reduce(IntWritable key, Iterator<IntWritable>
> values, OutputCollector<IntWritable, IntWritable>output, Reporter reporter)
> throws IOException
>             {
>                 int count = 0;
>                 while (values.hasNext())
>                 {
>                     count += values.next().get();
>                 }
>                 output.collect(key, new IntWritable(count));
>             }
>        }
>
>        public int run(String[] args) throws Exception
>        {
>             Configuration conf = getConf();
>
>             JobConf job = new JobConf(conf, CitationHistogram.class);
>
>             Path in = new Path(args[0]);
>             Path out = new Path(args[1]);
>             FileInputFormat.setInputPaths(job, in);
>             FileOutputFormat.setOutputPath(job, out);
>
>             job.setJobName("CitationHistogram");
>             job.setMapperClass(MapClass.class);
>             job.setReducerClass(Reduce.class);
>
>             job.setInputFormat(KeyValueTextInputFormat.class);
>             job.setOutputFormat(TextOutputFormat.class);
>             job.setOutputKeyClass(IntWritable.class);
>             job.setOutputValueClass(IntWritable.class);
>
>            JobClient.runJob(job);
>
>            return 0;
>        }
>
>       ......
>
> Both of the above code work on the same input file. So I felt weird. Could
> you please help me? Thanks so much!
>
> Best regards,
> Bing
>

Mime
View raw message