hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
Date Thu, 27 Aug 2009 15:33:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748405#action_12748405

Devaraj Das commented on MAPREDUCE-901:

bq. 1. I'd replace the array of longs with explicit fields.
Owen, is this to avoid the array indexing? I kept it this way since the TaskCounter enum already
existed, and it seemed simple to have a one-to-one mapping from the enum to the array. Also,
in the future if we add another element to the TaskCounter enum, we don't have to explicitly
modify TaskMetrics since it would automatically handle that..

bq. 2. I'd drop the booleans for whether it has been used.
I kept this so as to retain the existing semantics where an unused counter is not displayed
to the user (for e.g., reduce-shuffle-bytes is not displayed as part of map tasks' counters).
But I am okay to do without the booleans..

bq. 3. The serialization should just be to write the fields in order using vints.
If we remove the booleans, then it will be just writing the long array elements which is already
written as vlongs... If we decide to switch to explicit fields for the longs then this part
would change slightly.. 

bq. 4. updateFrameworkCounters should just update the counters by using the enum values explicitly.
Again, as i pointed out in (1) above, it helps to iterate over TaskCounter enum.

bq. 5. updateFrameworkCounters should set, not increment the related counter.
Makes sense

bq. 6. TaskMetrics should be a field in the TaskStatus.
It already is. A little bit of cleanup may be required w.r.t passing the object around in
the TaskStatus methods..

> Move Framework Counters into a TaskMetric structure
> ---------------------------------------------------
>                 Key: MAPREDUCE-901
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.21.0
>         Attachments: 901_1.patch, 901_1.patch
> I think we should move all of the Counters that the framework updates into a single class
called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input
records, input bytes, output records, etc.
> It would both reduce the serialized size of the heartbeats (by shrinking the Counters
down to just the user's counters) and decrease the latency for updates to the JobTracker (since
Counters are sent at most 1/minute instead of 1/heartbeat).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message