hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paco <paco...@gmail.com>
Subject Combiner and KeyComposite
Date Sun, 04 Oct 2015 08:38:43 GMT
I am doing a secondary sort in Hadoop 2.6.0, I am following this 

I have the exact same code, but now I am trying to improve performance 
so I have decided to add a combiner. I have added two modifications:

Main file:


Combiner file:

|public class CombinerK extends Reducer<KeyWritable, KeyWritable, KeyWritable, KeyWritable>

     public void reduce(KeyWritable key, Iterator<KeyWritable> values, Context context)
throws IOException, InterruptedException {

         Iterator<KeyWritable> it = values;

         System.err.println("combiner " + key);

         KeyWritable first_value = it.next();
         System.err.println("va: " + first_value);

         while (it.hasNext()) {

             sum += it.next().getSs();

         context.write(key, first_value);


But it seems that it is not run because I can't find any logs file which 
have the word "combiner". When I saw counters after running, I could see:

|     Combine input records=4040000
     Combine output records=4040000

The combiner seems like it is being executed but it seems as it has been 
receiving a call for each key and by this reason it has the same number 
in input as output.

View raw message