hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raj Vishwanathan <rajv...@yahoo.com>
Subject Re: Measuring Shuffle time for MR job
Date Mon, 27 Aug 2012 13:54:37 GMT
You can extract the shuffle time from the job log.

Take a look at 

https://github.com/rajvish/hadoop-summary 


Raj



>________________________________
> From: Bertrand Dechoux <dechouxb@gmail.com>
>To: common-user@hadoop.apache.org 
>Sent: Monday, August 27, 2012 12:57 AM
>Subject: Re: Measuring Shuffle time for MR job
> 
>Shuffle time is considered as part of the reduce step. Without reduce,
>there is no need for shuffling.
>One way to measure it would be using the full reduce time with a
>'/dev/null' reducer.
>
>I am not aware of any way to measure it.
>
>Regards
>
>Bertrand
>
>On Mon, Aug 27, 2012 at 8:18 AM, praveenesh kumar <praveenesh@gmail.com>wrote:
>
>> Is there a way to know the total shuffle time of a map-reduce job - I mean
>> some command or output  that can tell that ?
>>
>> I want to measure total map, total shuffle and total reduce time for my MR
>> job -- how can I achieve it ? I am using hadoop 0.20.205
>>
>>
>> Regards,
>> Praveenesh
>>
>
>
>
>-- 
>Bertrand Dechoux
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message