hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-492) Global counters
Date Wed, 30 Aug 2006 22:48:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-492?page=comments#action_12431736 ] 
            
Milind Bhandarkar commented on HADOOP-492:
------------------------------------------

We need separate global counters because there is no way to programmatically add new metrics
to the TaskMetrics class, which will (after I resubmit my TaskMetrics patch) be accumulated
in JobTracker via heartbeats. OTOH, TaskMetrics could be implemented in terms of global counters.


> Global counters
> ---------------
>
>                 Key: HADOOP-492
>                 URL: http://issues.apache.org/jira/browse/HADOOP-492
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: arkady borkovsky
>
> It would be nice to have map / reduce job keep aggregated counts for arbitrary events
occuring in its tasks -- the numer of records processed, the numer of exceptions of a specific
type, the number of sentences in passive voice, whatever the jobs finds useful.
> This can be implemented by tasks periodically sending <name, value> pairs to the
jobtracker (in some implementations such messages are piggy-backed on the heartbeats), so
that the job tracker stores all the latests values from each task and aggregates them on a
request.  It should also make the aggregated values available at the job end.  The value for
a task would be flushed when the task fails.
> #491 and #490 may be related to this one.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message