hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ☼ R Nair (रविशंकर नायर) <ravishankar.n...@gmail.com>
Subject Re: Combiner and KeyComposite
Date Sun, 04 Oct 2015 08:42:05 GMT
Are you checking logs at correct place?

On Sun, Oct 4, 2015, 4:39 PM paco <pacopww@gmail.com> wrote:

> I am doing a secondary sort in Hadoop 2.6.0, I am following this tutorial:
> https://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/
>
> I have the exact same code, but now I am trying to improve performance so
> I have decided to add a combiner. I have added two modifications:
>
> Main file:
>
> job.setCombinerClass(CombinerK.class);
>
> Combiner file:
>
> public class CombinerK extends Reducer<KeyWritable, KeyWritable, KeyWritable, KeyWritable>
{
>
>     public void reduce(KeyWritable key, Iterator<KeyWritable> values, Context context)
throws IOException, InterruptedException {
>
>
>         Iterator<KeyWritable> it = values;
>
>         System.err.println("combiner " + key);
>
>         KeyWritable first_value = it.next();
>         System.err.println("va: " + first_value);
>
>         while (it.hasNext()) {
>
>             sum += it.next().getSs();
>
>         }
>         first_value.setS(sum);
>         context.write(key, first_value);
>
>
>     }
> }
>
> But it seems that it is not run because I can't find any logs file which
> have the word "combiner". When I saw counters after running, I could see:
>
>     Combine input records=4040000
>     Combine output records=4040000
>
> The combiner seems like it is being executed but it seems as it has been
> receiving a call for each key and by this reason it has the same number in
> input as output.
>

Mime
View raw message