hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bing Li <lbl...@gmail.com>
Subject IOException: Job Failed
Date Wed, 08 Feb 2012 22:44:55 GMT
Dear all,

When running some sample codes from Hadoop in Action, I got an IOException:
Job Failed.

Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at com.greatfree.testing.hadoop.CitationHistogram.run(Unknown Source)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.greatfree.testing.hadoop.CitationHistogram.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

For the following code, everything is fine.

        ......

        public static class MapClass extends MapReduceBase implements
Mapper<Text, Text, Text, Text>
        {
                public void map(Text key, Text value, OutputCollector<Text,
Text> output, Reporter reporter) throws IOException
                {
                        output.collect(value, key);
                }
        }

        public static class ReduceClass extends MapReduceBase implements
Reducer<Text, Text, Text, IntWritable>
        {
                public void reduce(Text key, Iterator<Text> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException
                {
                        int count = 0;
                        while (values.hasNext())
                        {
                                values.next();
                                count ++;
                        }
                        output.collect(key, new IntWritable(count));
                }
        }

        public int run(String[] args) throws Exception
        {
                Configuration conf = getConf();

                JobConf job = new JobConf(conf, MyJob.class);

                Path in = new Path(args[0]);
                Path out = new Path(args[1]);
                FileInputFormat.setInputPaths(job, in);
                FileOutputFormat.setOutputPath(job, out);

                job.setJobName("MyJobForCount");
                job.setMapperClass(MapClass.class);
                job.setReducerClass(ReduceClass.class);

                job.setInputFormat(KeyValueTextInputFormat.class);
                job.setOutputFormat(TextOutputFormat.class);
                job.setOutputKeyClass(Text.class);
                job.setOutputValueClass(Text.class);
                job.set("key.value.separator.in.input.line", ",");

                JobClient.runJob(job);
                return 0;
        }

        ......

However, when making some changes on the above code as follows, I got the
above exception.

        ......

        public static class MapClass extends MapReduceBase implements
Mapper<Text, Text, IntWritable, IntWritable>
        {
            private final static IntWritable uno = new IntWritable(1);
            private IntWritable citationCount = new IntWritable();

            public void map(Text key, Text value,
OutputCollector<IntWritable, IntWritable> output, Reporter reporter) throws
IOException
            {
                citationCount.set(Integer.parseInt(value.toString()));
                output.collect(citationCount, uno);
            }
        }

       public static class Reduce extends MapReduceBase implements
Reducer<IntWritable,IntWritable,IntWritable,IntWritable>
       {
            public void reduce(IntWritable key, Iterator<IntWritable>
values, OutputCollector<IntWritable, IntWritable>output, Reporter reporter)
throws IOException
            {
                int count = 0;
                while (values.hasNext())
                {
                    count += values.next().get();
                }
                output.collect(key, new IntWritable(count));
            }
       }

       public int run(String[] args) throws Exception
       {
            Configuration conf = getConf();

            JobConf job = new JobConf(conf, CitationHistogram.class);

            Path in = new Path(args[0]);
            Path out = new Path(args[1]);
            FileInputFormat.setInputPaths(job, in);
            FileOutputFormat.setOutputPath(job, out);

            job.setJobName("CitationHistogram");
            job.setMapperClass(MapClass.class);
            job.setReducerClass(Reduce.class);

            job.setInputFormat(KeyValueTextInputFormat.class);
            job.setOutputFormat(TextOutputFormat.class);
            job.setOutputKeyClass(IntWritable.class);
            job.setOutputValueClass(IntWritable.class);

           JobClient.runJob(job);

           return 0;
       }

      ......

Both of the above code work on the same input file. So I felt weird. Could
you please help me? Thanks so much!

Best regards,
Bing

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message