hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paco <paco...@gmail.com>
Subject Combiner and KeyComposite
Date Sun, 04 Oct 2015 08:38:43 GMT
I am doing a secondary sort in Hadoop 2.6.0, I am following this 
tutorial: 
https://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/

I have the exact same code, but now I am trying to improve performance 
so I have decided to add a combiner. I have added two modifications:

Main file:

|job.setCombinerClass(CombinerK.class);
|

Combiner file:

|public class CombinerK extends Reducer<KeyWritable, KeyWritable, KeyWritable, KeyWritable>
{

     public void reduce(KeyWritable key, Iterator<KeyWritable> values, Context context)
throws IOException, InterruptedException {


         Iterator<KeyWritable> it = values;

         System.err.println("combiner " + key);

         KeyWritable first_value = it.next();
         System.err.println("va: " + first_value);

         while (it.hasNext()) {

             sum += it.next().getSs();

         }
         first_value.setS(sum);
         context.write(key, first_value);


     }
}
|

But it seems that it is not run because I can't find any logs file which 
have the word "combiner". When I saw counters after running, I could see:

|     Combine input records=4040000
     Combine output records=4040000
|

The combiner seems like it is being executed but it seems as it has been 
receiving a call for each key and by this reason it has the same number 
in input as output.


Mime
View raw message