hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <omal...@apache.org>
Subject Re: Measuring running times
Date Wed, 17 Mar 2010 15:45:48 GMT

On Mar 17, 2010, at 4:47 AM, Antonio D'Ettole wrote:

> Hi everybody,
> as part of my project work at school I'm running some Hadoop jobs on a
> cluster. I'd like to measure exactly how long each phase of the  
> process
> takes: mapping, shuffling (ideally divided in copying and sorting) and
> reducing.

Look at the job history logs. They break down the times for each task.  
You need to run a script to aggregate them. You can see an example of  
the aggregation on my petabyte sort description:

http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html

-- Owen

Mime
View raw message