hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2124) Add job counters for measuring time spent in three different phases in reducers
Date Mon, 11 Oct 2010 18:39:35 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919940#action_12919940
] 

Scott Chen commented on MAPREDUCE-2124:
---------------------------------------

Hey Chris,

We can put this in job counter (the one in JobInProgress) instead of task counter.
So these counters will be recorded per job. The memory cost should be very minor.
Also the cost for collecting them is very low. We can do this in JobInProgress.meterTaskAttempt()
which will be called only once after an attempt is finished.

These times are recorded in job history but it will be nice to have them on the web UI if
there is no much cost.
What do you think?

Scott

> Add job counters for measuring time spent in three different phases in reducers
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2124
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2124
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Minor
>             Fix For: 0.22.0
>
>
> We currently have SLOTS_MILLIS_REDUCES which measures the total slot time of reducer.
> It will be useful if we have
> {code}
> SLOTS_MILLIS_REDUCES_COPY
> SLOTS_MILLIS_REDUCES_SORT
> SLOTS_MILLIS_REDUCES_REDUCE
> {code}
> which measures three different phases of a reducer.
> This will help us identify the bottleneck of the reducers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message