hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-481) Hadoop mapred metrics should include per job input/output statistics rather than per-task statistics
Date Tue, 05 Sep 2006 18:29:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-481?page=comments#action_12432647 ] 
            
Doug Cutting commented on HADOOP-481:
-------------------------------------

Shouldn't we use our existing metrics API for stuff like this?  As with HADOOP-492, it seems
like the TaskTracker and JobTracker should implement the MetricsContext API, providing a MetricsRecord
factory.  These can be used by the MapReduce kernel code for the metrics desired here, and
in supplied to user code the uses in HADOOP-492.

We might even need to write a multiplexing MetricsContext, that can send metrics to both the
JobTracker and to, e.g., Ganglia.  But we should not be adding new metrics APIs when we already
have one.  If the current metrics API is somehow inappropriate, let's fix that instead of
create another.

> Hadoop mapred metrics should include per job input/output statistics rather than per-task
statistics
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-481
>                 URL: http://issues.apache.org/jira/browse/HADOOP-481
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred, metrics
>    Affects Versions: 0.6.0
>            Reporter: Milind Bhandarkar
>         Assigned To: Milind Bhandarkar
>            Priority: Minor
>             Fix For: 0.6.0
>
>         Attachments: reports.patch
>
>
> Currently hadoop reports metrics such as input bytes, input records, etc on per-task
basis. Accurate aggregation of these metrics is required at the job-level and reporting should
be done on a per-job basis.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message