hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sangroya <sangroyaa...@gmail.com>
Subject Re: Measuring time in a Hadoop MapReduce job.
Date Mon, 04 Jul 2011 09:45:46 GMT
Dear Keren,

Thanks for your response!

I understood from your response that I can parse the job history file
and extract the information about START_TIME and FINISH_TIME.

Would it be possible to know about a simpler way where I know a
MAP_PHASE_START_TIME and MAP_PHASE_FINISH_TIME and similarly
REDUCE_PHASE_START_TIME and REDUCE_PHASE_FINISH_TIME.


Thanks,
Amit



On Mon, Jul 4, 2011 at 11:12 AM, Keren Ouaknine <kereno@gmail.com> wrote:
> Hello Amit,
>
> There are several ways, the one I am currently using for measurements is
> based on the job history file.
> This file has a line for each event, thus the last map event (indicating on
> the completion of the map task) would be the last record in time with both
> string "MAP" and "TASK_FINISHED". Similarly for the last reduce event: look
> for the last record in time with both string "REDUCE" and "TASK_FINISHED".
>
> If you find a better way, please update me!
>
> Thanks,
> Keren
>
>
> On Mon, Jul 4, 2011 at 12:04 PM, Amit Sangroya <sangroyaamit@gmail.com>wrote:
>
>> Hi,
>>
>> I am trying to monitor the time to complete a map phase and reduce
>> phase in hadoop. Is there any way to measure the time taken to
>> complete map and reduce phase in a cluster.
>>
>> Thanks,
>> Amit
>>
>
>
>
> --
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
>

Mime
View raw message