hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2125) Put map-reduce framework counters to JobTrackerMetricsInst
Date Tue, 02 Nov 2010 00:42:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927239#action_12927239

Scott Chen commented on MAPREDUCE-2125:

Hey Luke,

What's the job completion rate on your cluster? If you just run a few big jobs that takes
a while, It probably doesn't matter. But if you have jobs that takes seconds to minutes rather
than hours (typically submitted via automated systems and getCounters is not even called because
they're intermediate steps) the job completion rate can be very high, I've seen over 100 per
second on our clusters.

If the use case is for many small jobs, each getCounter() call will be cheap. So in this case
it will still be OK.
I think the key here is that this change only add one more look at all task counter. So from
the throughput point of view, it is not that large.  

Our job completion rate is about 20 jobs/minute in average.
Comparing to the general use case in our cluster that HIVE pulls the jobCounters periodically,
this one is very light weight because it does getCounters only once.

> Put map-reduce framework counters to JobTrackerMetricsInst
> ----------------------------------------------------------
>                 Key: MAPREDUCE-2125
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2125
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-2125.txt
> We have lots of useful information in the framework counters including #spills, filesystem
read and write.
> It will be nice to put them all in the jobtracker metrics to get a global view of all
these numbers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message