hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kishore <alajangikish...@gmail.com>
Subject Re: question about combiner
Date Fri, 10 May 2013 18:00:52 GMT
Combiner will be used between mapper and reduce, so the mapper output for both with used combiner
and without used combiner are same.

Thanks,
Kishore.

Sent from my iPhone

On 10-May-2013, at 8:49 PM, Han JU <ju.han.felix@gmail.com> wrote:

> Hi,
> 
> For a MapReduce job with lots of intermediate results between mapper and reducer, I implement
a combiner function with a more compact representation of the result data and I verified the
final result is good when using combiner. But when I look at the job counter "FILE_BYTES_WRITTEN"
or "Reduce shuffle bytes", the number with combiner is twice bigger than without combiner.
In my comprehension, these two counters represent the output size of mapper. And with a combiner,
the size of mapper output should decrease, but it's not the case here.
> 
> So it means that my combiner doesn't work and it actually increase the size of mapper
output? 
> 
> Thanks!
> -- 
> JU Han
> 
> Software Engineer Intern @ KXEN Inc.
> UTC   -  Université de Technologie de Compiègne
>      GI06 - Fouille de Données et Décisionnel
> 
> +33 0619608888

Mime
View raw message