hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3136) Assign multiple tasks per TaskTracker heartbeat
Date Fri, 01 Aug 2008 03:00:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618922#action_12618922

Joydeep Sen Sarma commented on HADOOP-3136:

i don't know how big a change we are envisaging here. my suggestion was thinking if we wanted
to minimize changes to current code base. I don't think the protocol suggested would send
any more heartbeats than we do in the worst case today (or that at least it could be tuned
to get that property) - but if the plan going forward is to reduce heartbeats - then that's
not good enough.

If we are planning non-trivial changes - it seems to me that it would be nice as a first step
to have scheduling decisions dispatched pro-actively to slaves on new job arrival (after considering
global resource availability). that itself should save on latency of scheduling on an idling
cluster. on a backloaded cluster, scheduling reactively to heartbeats seems reasonable (in
fact - on a backloaded cluster - it's likely that each heartbeat would only advertize one
available slot. unless of course we move away from current protocol for sending heartbeat
on task completion).

the other significant improvement i can think of is maintaining a runnable task queue on each
TT (in addition to the ones running). that way there would be no idle time lost to getting
new tasks. the protocol of dispatching tasks to TT runnable queues could happen in the background
where one could make optimizations to reduce network traffic. (for example - if the TT had
a good sized queue of runnable tasks then it could afford to delay sending heartbeats on task
completion (since delaying would no longer cause idle time)). This may also allow the JT to
make batched scheduling decisions (instead of immediately looking for something to schedule
when slots become free - the JT could wait and accumulate some larger number of free slots
before making a global scheduling decision over all those slots and the available tasks. this
might make things more globally optimal while still not causing idle time (since there every
TT has a queue of runnable tasks)).

> Assign multiple tasks per TaskTracker heartbeat
> -----------------------------------------------
>                 Key: HADOOP-3136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3136
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>             Fix For: 0.19.0
> In today's logic of finding a new task, we assign only one task per heartbeat.
> We probably could give the tasktracker multiple tasks subject to the max number of free
slots it has - for maps we could assign it data local tasks. We could probably run some logic
to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks,
tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we
could give it reduce task(s) as well. Again for reduces we could probably run some logic to
give more tasks to nodes that are closer to nodes running most maps (assuming data generated
is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, and
we know that most maps are data/rack local, we try to schedule ~70% of the reducers there.
> Thoughts?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message