hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "arkady borkovsky (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-492) Global counters
Date Tue, 29 Aug 2006 21:35:24 GMT
Global counters

                 Key: HADOOP-492
                 URL: http://issues.apache.org/jira/browse/HADOOP-492
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: arkady borkovsky

It would be nice to have map / reduce job keep aggregated counts for arbitrary events occuring
in its tasks -- the numer of records processed, the numer of exceptions of a specific type,
the number of sentences in passive voice, whatever the jobs finds useful.

This can be implemented by tasks periodically sending <name, value> pairs to the jobtracker
(in some implementations such messages are piggy-backed on the heartbeats), so that the job
tracker stores all the latests values from each task and aggregates them on a request.  It
should also make the aggregated values available at the job end.  The value for a task would
be flushed when the task fails.

#491 and #490 may be related to this one.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message