hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Willis <swil...@compete.com>
Subject RE: Reducer not firing
Date Tue, 17 Apr 2012 20:19:59 GMT
Try putting @Override before your reduce method to make sure you're overriding the method properly.
You'll get a compile time error if not.

-Steven Willis

From: Bejoy KS [mailto:bejoy.hadoop@gmail.com]
Sent: Tuesday, April 17, 2012 10:03 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Reducer not firing

Hi Akro
>From the naming of output files, your job has the reduce phase. But the reducer being
used is the IdentityReducer instead of your custom reducer. That is the reason you are seeing
the same map output in the output files as well. You need to evaluate your code and logs to
see why IdentityReducer is being triggered.
Bejoy KS

Sent from handheld, please excuse typos.
From: kasi subrahmanyam <kasisubbu440@gmail.com<mailto:kasisubbu440@gmail.com>>
Date: Tue, 17 Apr 2012 19:10:33 +0530
To: <mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>>
ReplyTo: mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>
Subject: Re: Reducer not firing

Could you comment the property where you are setting the number of reducer tasks and see the
behaviour of the program once.
If you already tried could you share the output
On Tue, Apr 17, 2012 at 3:00 PM, Devaraj k <devaraj.k@huawei.com<mailto:devaraj.k@huawei.com>>
Can you check the task attempt logs in your cluster and find out what is happening in the
reduce phase. By default task attempt logs present in $HADOOP_LOG_DIR/userlogs/<job-id>/.
There could be some bug exist in your reducer which is leading to this output.


From: Arko Provo Mukherjee [arkoprovomukherjee@gmail.com<mailto:arkoprovomukherjee@gmail.com>]
Sent: Tuesday, April 17, 2012 2:07 PM
To: mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>
Subject: Re: Reducer not firing


Many thanks for the reply.

The 'no_of_reduce_tasks' is set to 2. I have a print statement before
the code I pasted below to check that.

Also I can find two output files part-r-00000 and part-r-00001. But
they contain the values that has been outputted by the Mapper logic.

Please let me know what I can check further.

Thanks a lot in advance!

Warm regards

On Tue, Apr 17, 2012 at 12:48 AM, Devaraj k <devaraj.k@huawei.com<mailto:devaraj.k@huawei.com>>
> Hi Arko,
>    What is value of  'no_of_reduce_tasks'?
> If no of reduce tasks are 0, then the map task will directly write map output  into the
Job output path.
> Thanks
> Devaraj
> ________________________________________
> From: Arko Provo Mukherjee [arkoprovomukherjee@gmail.com<mailto:arkoprovomukherjee@gmail.com>]
> Sent: Tuesday, April 17, 2012 10:32 AM
> To: mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>
> Subject: Reducer not firing
> Dear All,
> I am porting code from the old API to the new API (Context objects)
> and run on Hadoop 0.20.203.
> Job job_first = new Job();
> job_first.setJarByClass(My.class);
> job_first.setNumReduceTasks(no_of_reduce_tasks);
> job_first.setJobName("My_Job");
> FileInputFormat.addInputPath( job_first, new Path (Input_Path) );
> FileOutputFormat.setOutputPath( job_first, new Path (Output_Path) );
> job_first.setMapperClass(Map_First.class);
> job_first.setReducerClass(Reduce_First.class);
> job_first.setMapOutputKeyClass(IntWritable.class);
> job_first.setMapOutputValueClass(Text.class);
> job_first.setOutputKeyClass(NullWritable.class);
> job_first.setOutputValueClass(Text.class);
> job_first.waitForCompletion(true);
> The problem I am facing is that instead of emitting values to
> reducers, the mappers are directly writing their output in the
> OutputPath and the reducers and not processing anything.
> As read from the online materials that are available both my Map and
> Reduce method uses the context.write method to emit the values.
> Please help. Thanks a lot in advance!!
> Warm regards
> Arko

View raw message