hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10090) Jobtracker metrics not updated properly after execution of a mapreduce job
Date Tue, 12 Nov 2013 20:38:17 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820421#comment-13820421

Luke Lu commented on HADOOP-10090:

I was aware of the suboptimal behavior and hoped it'd be OK for metrics, which don't require
strong consistency.

#1 incurs unnecessary overhead (updating jmx cache) for people who don't use JMX. This is
the reason of the current cache logic.
#2 is risky, as we don't know all existing jmx query patterns (especially due to HDFS-5333).
User (admins) actually already has a choice to use a small JMX cache TTL for refreshness.

How about #3: we only initialize and update the JMX cache when JMX is first used and stops
updating after a period inactivity and reinitialize and update JMX cache upon activity. Initialize/reinitialize
is a "dense" update, while "update" means the "sparse" update with the current lastRecs mechanisms.
I think #3 is should be a fairly straightforward patch and more flexible than #1 and #2.

> Jobtracker metrics not updated properly after execution of a mapreduce job
> --------------------------------------------------------------------------
>                 Key: HADOOP-10090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 1.2.1
>            Reporter: Ivan Mitic
>            Assignee: Ivan Mitic
>         Attachments: OneBoxRepro.png
> After executing a wordcount mapreduce sample job, jobtracker metrics are not updated
properly. Often times the response from the jobtracker has higher number of job_completed
than job_submitted (for example 8 jobs completed and 7 jobs submitted). 
> Issue reported by Toma Paunovic.

This message was sent by Atlassian JIRA

View raw message