hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3136) Assign multiple tasks per TaskTracker heartbeat
Date Wed, 17 Sep 2008 05:59:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631662#action_12631662
] 

Arun C Murthy commented on HADOOP-3136:
---------------------------------------

Ok, we really need to get better scheduling to go along with assigning multiple tasks -- blindly
assigning multiple tasks, per the current patch, has a very negative effect on locality of
tasks as TaskTrackers 'steal' data-local/rack-local tasks from each other and consequently
adversely affects jobs. This patch slowed GridMix down by around 30%.

Here are some hallway ideas we threw around today for _better_ scheduling which is imperative
for assigning multiple tasks:

1. Oversubscribe Slots

  In this approach we need to assign more than one task per slot and queue them up at the
TaskTracker. This approach allows for much larger heartbeat intervals (the TaskTracker has
queued-up work to finish) and helps with scaling out clusters.

2. Pre-allocation of tasks to TaskTrackers via Global Scheduling

  Here we build up queues for _each_ TaskTracker (and each Rack) at the JobTracker based on
locality. Each task might be on multiple queues depending on which DataNode/TaskTracker it's
data is present. When a TaskTracker advertises empty slots we just pick off it's list and
assign it. Basically this implies that we consider the 'global' picture during scheduling
and ensures that TaskTrackers do not 'steal' data-local tasks from each other. 

  Of course to ensure that the highest-priority job doesn't get _starved_ we need to assign
atleast one of it's tasks on each TaskTracker's heartbeat, even if it means we schedule an
off-rack task. Similarly, to avoid spreading task-allocation too thin i.e. across too many
jobs, we need to ensure that that the TaskTrackers' lists only contain tasks from a reasonably
small set of the highest-priority jobs.

3. HADOOP-2014: Scheduling off-rack tasks 

----

Thoughts?

> Assign multiple tasks per TaskTracker heartbeat
> -----------------------------------------------
>
>                 Key: HADOOP-3136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3136
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-3136_0_20080805.patch, HADOOP-3136_1_20080809.patch, HADOOP-3136_2_20080911.patch
>
>
> In today's logic of finding a new task, we assign only one task per heartbeat.
> We probably could give the tasktracker multiple tasks subject to the max number of free
slots it has - for maps we could assign it data local tasks. We could probably run some logic
to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks,
tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we
could give it reduce task(s) as well. Again for reduces we could probably run some logic to
give more tasks to nodes that are closer to nodes running most maps (assuming data generated
is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, and
we know that most maps are data/rack local, we try to schedule ~70% of the reducers there.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message