hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Which component manages slow tasks?
Date Sat, 09 Jun 2012 13:38:49 GMT

You seem to be looking for the speculative execution code?

For MR1, in 1.x or 0.20.x and below, you can find that in the
JobInProgress#findNewMapTask method (go down to the bit which does
speculative execution logic for maps and reduces depending on if the
job enabled it for itself).

For MR2, in 2.x and above, since it offers a way to have pluggable
speculative algorithms, the default implementation (a slightly
advanced version of the one in 1.0) is available in the
DefaultSpeculator class (under the mapreduce-client-app package). This
is used by the MRAppMaster (and is event driven).

If you are instead asking about the task timeout monitor (that kills
tasks that haven't reported for a period):

For MR1 it is in TaskTracker#markUnresponsiveTasks method (that uses
timeout values set per task (from a job)).

For MR2 similar code is in TaskHeartbeatHandler#PingChecker thread
class. This is again in the mapreduce-client-app package.

Preemption is a different operation than monitoring slow jobs/tasks.

I believe you should be able to, with any decent IDE, follow the above
entry points and then be able to drill up/down from there to analyze
the code. In case you get stuck somewhere in understanding something
around these features, feel free to ask us here!

On Fri, Jun 8, 2012 at 9:14 PM, Merto Mertek <masmertoz@gmail.com> wrote:
> I am interested in which component of hadoop and how is managed slow task
> execution.
> Is this a tasktracker, jobtracker or a scheduler?
> In fairscheduling (0.20.205) there exist a preemption mechanism but it just
> handles min and fairshares and kills last submited tips..
> Thanks.

Harsh J

View raw message