hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
Date Tue, 28 Sep 2010 22:45:38 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Luke Lu updated MAPREDUCE-901:
------------------------------

    Attachment: FrameworkCounterGroup.java

Looking through the patches, it seems that the existing approach requires sweeping changes
to internal apis across pretty much all mapreduce components.

I'm proposing a new approach to this issue: refactor the counter framework (while keeping
the existing api) so that it's possible to have different implementation of counter groups,
one of which is the FrameworkCounterGroup (see attached file.)

This way, we can achieve the benefit of more efficient implementation for framework counters
without changing the client code.

The problem is a little complicated due to need to support of both old and new counters interface
(both marked public and stable.) I'm working on a more complete patch to minimize the code
duplication between new and old counter code. But I'm confident that the resulting code will
be more concise and general (to support any future framework counters in different groups.)
than the existing approach.

> Move Framework Counters into a TaskMetric structure
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-901
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Luke Lu
>         Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch,
MAPREDUCE-901.patch
>
>
> I think we should move all of the Counters that the framework updates into a single class
called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input
records, input bytes, output records, etc.
> It would both reduce the serialized size of the heartbeats (by shrinking the Counters
down to just the user's counters) and decrease the latency for updates to the JobTracker (since
Counters are sent at most 1/minute instead of 1/heartbeat).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message