hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bertrand Dechoux <decho...@gmail.com>
Subject Re: Measuring Shuffle time for MR job
Date Mon, 27 Aug 2012 07:57:05 GMT
Shuffle time is considered as part of the reduce step. Without reduce,
there is no need for shuffling.
One way to measure it would be using the full reduce time with a
'/dev/null' reducer.

I am not aware of any way to measure it.



On Mon, Aug 27, 2012 at 8:18 AM, praveenesh kumar <praveenesh@gmail.com>wrote:

> Is there a way to know the total shuffle time of a map-reduce job - I mean
> some command or output  that can tell that ?
> I want to measure total map, total shuffle and total reduce time for my MR
> job -- how can I achieve it ? I am using hadoop 0.20.205
> Regards,
> Praveenesh

Bertrand Dechoux

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message