hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: MAP_INPUT_RECORDS counter in the reducer
Date Tue, 17 Sep 2013 13:49:06 GMT
In the normal configuration, the issue here is that Reducers can start
before all the Maps have finished so it is not possible to get the number
(or make sense of it even if you are able to,)

Having said that, you can specifically make sure that Reducers don't start
until all your maps have completed. It will of course slow down your job. I
don't know whether with this option it will work or not, but you can try
(until experts have some advise already.)

Regards,
Shahab


On Tue, Sep 17, 2013 at 6:09 AM, Yaron Gonen <yaron.gonen@gmail.com> wrote:

> Hi,
> Is there a way for the reducer to get the total number of input records to
> the map phase?
> For example, I want the reducer to normalize a sum by dividing it in the
> number of records. I tried getting the value of that counter by using the
> line:
>
> context.getCounter(Task.Counter.MAP_INPUT_RECORDS).getValue();
>
> in the reducer code, but I got 0.
>
> Thanks!
> Yaron
>

Mime
View raw message