hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
Date Fri, 01 Oct 2010 19:23:38 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Luke Lu updated MAPREDUCE-901:
------------------------------

    Attachment: mr-901-trunk-v1.patch

v1 patch passes all existing counters tests, probably good enough to bother hudson.

This is patch is basically a refactor of the counters framework. Blame java generics for the
syntax :)

Highlight of the patch:
* No (intentional) user API changes.
* Make mapred.Counters.Counter (legacy) and mapreduce.Counter (new) abstract
** the existing constructors are already package private, needed to fix one test to use proper
API instead of 'new'
* Make mapred.Counters.Group (legacy) and mapreduce.CounterGroup interface
** the existing constructors are already package private, needed to fix EventReader to use
proper API instead of 'new'
* Implement AbstractCounters and generic counter group in mapreduce.AbstractCounterGroup and
framework counter group in mapreduce.FrameworkCounterGroup
** Framework group counters have efficient in memory (fixed arrays of long and light weight
counter facade) and serialized ((vint ordinal, vint value)* tuples) representations. Framework
counters can be easily declared in CounterGroupFactory.
* mapred.Counters and mapreduce.Counters contain the delta of AbstractCounters and use mixins
to adapt to different interface
** Much of the generics dancing is to support type safe Iterable in both legacy and new Counters
interface.

> Move Framework Counters into a TaskMetric structure
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-901
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Luke Lu
>         Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch,
MAPREDUCE-901.patch, mr-901-trunk-v1.patch
>
>
> I think we should move all of the Counters that the framework updates into a single class
called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input
records, input bytes, output records, etc.
> It would both reduce the serialized size of the heartbeats (by shrinking the Counters
down to just the user's counters) and decrease the latency for updates to the JobTracker (since
Counters are sent at most 1/minute instead of 1/heartbeat).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message