hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2125) Put map-reduce framework counters to JobTrackerMetricsInst
Date Mon, 01 Nov 2010 20:47:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927147#action_12927147
] 

Luke Lu commented on MAPREDUCE-2125:
------------------------------------

bq. You are right. The getCounter() will happen in completeJob(). But it is called once per
job for the entire life-cycle of one job. So it has very minor impact on JT performance.

What's the job completion rate on your cluster? If you just run a few big jobs that takes
a while, It probably doesn't matter. But if you have jobs that takes seconds to minutes rather
than hours (typically submitted via automated systems and getCounters is not even called because
they're intermediate steps) the job completion rate can be very high, I've seen over 100 per
second on our clusters.

I'd be more comfortable with the patch, if we have an easy way to disable the expensive metrics.
One way to do it without a lot of ifs is to create an alternative instrumentation class that
subclass from the default class.

> Put map-reduce framework counters to JobTrackerMetricsInst
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-2125
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2125
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-2125.txt
>
>
> We have lots of useful information in the framework counters including #spills, filesystem
read and write.
> It will be nice to put them all in the jobtracker metrics to get a global view of all
these numbers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message