hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2125) Put map-reduce framework counters to JobTrackerMetricsInst
Date Mon, 01 Nov 2010 19:04:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927090#action_12927090
] 

Luke Lu commented on MAPREDUCE-2125:
------------------------------------

There are two major issues with the proposal/patch.

# The necessity of having these total aggregate counts in real time. Rumen or other MR log
processing tools can get these aggregates for performance analysis without impacting JT performance.
# Have you tried to benchmark the patch at scale? Calling job.getCounters in completeJob would
bring down a busy JT on a large cluster to its knee. Think about calling getCounters (which
is essentially a O(n) operation) a few hundred times per second!

If you really want these counters in real time, you should implement it in TT where it can
send the metrics to distributed metrics aggregators with UDP etc. and can be easily disabled/enabled
via the metrics system.

> Put map-reduce framework counters to JobTrackerMetricsInst
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-2125
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2125
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-2125.txt
>
>
> We have lots of useful information in the framework counters including #spills, filesystem
read and write.
> It will be nice to put them all in the jobtracker metrics to get a global view of all
these numbers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message