hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: Which Subphases Do Times on JobHistory Web UI Cover
Date Wed, 25 Sep 2013 02:17:40 GMT
Average map time includes everything the map task is doing, i.e. all the
things you mentioned.  Reduce time does not cover shuffle time.  Reduce
time is the time spent calling the reducer function and writing its output
to HDFS.  Merge time is related to reduce, not map.

-Sandy


On Tue, Sep 24, 2013 at 6:57 PM, Efe Gencer <efegencer@gmail.com> wrote:

> *By the way this question is about Apache Hadoop Release 2.1.0-beta.
>
> Thanks,
>
>
>
> 2013/9/24 Efe Gencer <efegencer@gmail.com>
>
>> Hi All,
>>
>> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
>> ...
>> Started: <start time>
>> Finished: <finish time>
>> Elapsed: 12 mins, 5sec
>> Diagnostics:
>> *Average Map Time*: 1 mins, 40 sec
>> Average Reduce Time: 12 sec
>> Average Shuffle Time: 10 mins, 8 sec
>> Average Merge Time: 1 sec
>> ...
>>
>> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
>> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
>> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
>> to hdfs, etc)
>> 3) Is Average Merge time related with map or reduce? (since they both
>> have merge phases)
>>
>> Best,
>> Efe
>>
>
>

Mime
View raw message