hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5499) JobTracker metrics do not match job counters
Date Mon, 16 Mar 2009 11:40:51 GMT
JobTracker metrics do not match job counters
--------------------------------------------

                 Key: HADOOP-5499
                 URL: https://issues.apache.org/jira/browse/HADOOP-5499
             Project: Hadoop Core
          Issue Type: Bug
          Components: mapred, metrics
    Affects Versions: 0.20.0
            Reporter: Chris Douglas


After one run of a 10 map, 3 reduce sort job, the mapred metrics report:
{noformat}
mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
  reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
  maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
{noformat}

After the second (w/ one manually killed reduce)
{noformat}
mapred.tasktracker: hostName=pineapple-lm, sessionId=, mapTaskSlots=3, maps_running=0, \
   reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
mapred.jobtracker: hostName=pineapple-lm, sessionId=, jobs_completed=2, jobs_submitted=2,
\
  maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
{noformat}

The counters report the expected 10/3 map/reduce completions, the second job reporting a single
failure.

# The {{maps_launched}} and {{reduces_launched}} counts are likely recording setup and cleanup
tasks as well
# After being recorded among launched tasks, it looks like setup and cleanup are _not_ included
among completed tasks
# {{tasks_completed}} on the TaskTracker should include only user tasks
# The {{reduces_launched}} count makes little sense to me. With three reduces launched per
job and one failed, what's the other launched reduce counting?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message