hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-2047) reduce overhead of findSpeculativeTask
Date Tue, 31 Aug 2010 22:10:53 GMT
reduce overhead of findSpeculativeTask

                 Key: MAPREDUCE-2047
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2047
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobtracker
         Environment: hadoop-20 with HADOOP-2141
            Reporter: Joydeep Sen Sarma
            Assignee: Joydeep Sen Sarma

We are bottlenecked (in the JT) on the jobtracker lock and calls to findSpeculativeTask frequently
show up as one of the top routines (by time) called holding this lock.

this routine calls canBeSpeculated() and hasRunOnMachine() for each task in a candidate job.
Both these routines are reasonably expensive when invoked repeatedly  for thousands of tasks.
The top candidates for speculation from a job only need to be refreshed periodically (and
not once every heartbeat) - and we can can avoid most of these invocations this way.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message