hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Efe Gencer <efegen...@gmail.com>
Subject Re: Which Subphases Do Times on JobHistory Web UI Cover
Date Wed, 25 Sep 2013 01:57:51 GMT
*By the way this question is about Apache Hadoop Release 2.1.0-beta.

Thanks,


2013/9/24 Efe Gencer <efegencer@gmail.com>

> Hi All,
>
> In JobHistory Web UI under Job > "Map Tasks" I see something as follows:
> ...
> Started: <start time>
> Finished: <finish time>
> Elapsed: 12 mins, 5sec
> Diagnostics:
> *Average Map Time*: 1 mins, 40 sec
> Average Reduce Time: 12 sec
> Average Shuffle Time: 10 mins, 8 sec
> Average Merge Time: 1 sec
> ...
>
> 1) I wonder which sub-map phases "Map Time" contains (e.g. map function,
> sort, spill, merge, read and transfer corresponding filesplit from HDFS)
> 2) Does Reduce time covers Shuffle Time? What else does it cover? (write
> to hdfs, etc)
> 3) Is Average Merge time related with map or reduce? (since they both have
> merge phases)
>
> Best,
> Efe
>

Mime
View raw message