Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Message-ID: <5610E593.9090502@gmail.com>
Date: Sun, 04 Oct 2015 10:38:43 +0200
From: paco <pacopww@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Icedove/31.8.0
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: Combiner and KeyComposite
Content-Type: multipart/alternative;
 boundary="------------000504070302080505030706"

This is a multi-part message in MIME format.
--------------000504070302080505030706
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

I am doing a secondary sort in Hadoop 2.6.0, I am following this 
tutorial: 
https://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/

I have the exact same code, but now I am trying to improve performance 
so I have decided to add a combiner. I have added two modifications:

Main file:

|job.setCombinerClass(CombinerK.class);
|

Combiner file:

|public class CombinerK extends Reducer<KeyWritable, KeyWritable, KeyWritable, KeyWritable> {

     public void reduce(KeyWritable key, Iterator<KeyWritable> values, Context context) throws IOException, InterruptedException {


         Iterator<KeyWritable> it = values;

         System.err.println("combiner " + key);

         KeyWritable first_value = it.next();
         System.err.println("va: " + first_value);

         while (it.hasNext()) {

             sum += it.next().getSs();

         }
         first_value.setS(sum);
         context.write(key, first_value);


     }
}
|

But it seems that it is not run because I can't find any logs file which 
have the word "combiner". When I saw counters after running, I could see:

|     Combine input records=4040000
     Combine output records=4040000
|

The combiner seems like it is being executed but it seems as it has been 
receiving a call for each key and by this reason it has the same number 
in input as output.


--------------000504070302080505030706
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="post-text" itemprop="text">
      <p>I am doing a secondary sort in Hadoop 2.6.0, I am following
        this tutorial:
        <a
href="https://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/"
          rel="nofollow">https://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/</a></p>
      <p>I have the exact same code, but now I am trying to improve
        performance so I have decided to add a combiner. I have added
        two modifications:</p>
      <p>Main file:</p>
      <pre><code>job.setCombinerClass(CombinerK.class);
</code></pre>
      <p>Combiner file:</p>
      <pre><code>public class CombinerK extends Reducer&lt;KeyWritable, KeyWritable, KeyWritable, KeyWritable&gt; {

    public void reduce(KeyWritable key, Iterator&lt;KeyWritable&gt; values, Context context) throws IOException, InterruptedException {


        Iterator&lt;KeyWritable&gt; it = values;

        System.err.println("combiner " + key);

        KeyWritable first_value = it.next();
        System.err.println("va: " + first_value);

        while (it.hasNext()) {

            sum += it.next().getSs();

        }
        first_value.setS(sum);
        context.write(key, first_value);


    }
}
</code></pre>
      <p>But it seems that it is not run because I can't find any logs
        file which have the word "combiner". When I saw counters after
        running, I could see:</p>
      <pre><code>    Combine input records=4040000
    Combine output records=4040000
</code></pre>
      <p>The combiner seems like it is being executed but it seems as it
        has been receiving a call for each key and by this reason it has
        the same number in input as output.</p>
    </div>
  </body>
</html>

--------------000504070302080505030706--