hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-4665) Add preemption to the fair scheduler
Date Sat, 15 Nov 2008 23:52:44 GMT
Add preemption to the fair scheduler

                 Key: HADOOP-4665
                 URL: https://issues.apache.org/jira/browse/HADOOP-4665
             Project: Hadoop Core
          Issue Type: New Feature
          Components: contrib/fair-share
            Reporter: Matei Zaharia

Task preemption is necessary in a multi-user Hadoop cluster for two reasons: users might submit
long-running tasks by mistake (e.g. an infinite loop in a map program), or tasks may be long
due to having to process large amounts of data. The Fair Scheduler (HADOOP-3746) has a concept
of guaranteed capacity for certain queues, as well as a goal of providing good performance
for interactive jobs on average through fair sharing. Therefore, it will support preempting
under two conditions:
1) A job isn't getting its _guaranteed_ share of the cluster for at least T1 seconds.
2) A job is getting significantly less than its _fair_ share for T2 seconds (e.g. less than
half its share).

T1 will be chosen smaller than T2 (and will be configurable per queue) to meet guarantees
quickly. T2 is meant as a last resort in case non-critical jobs in queues with no guaranteed
capacity are being starved.

When deciding which tasks to kill to make room for the job, we will use the following heuristics:
- Look for tasks to kill only in jobs that have more than their fair share, ordering these
by deficit (most overscheduled jobs first).
- For maps: kill tasks that have run for the least amount of time (limiting wasted time).
- For reduces: similar to maps, but give extra preference for reduces in the copy phase where
there is not much map output per task (at Facebook, we have observed this to be the main time
we need preemption - when a job has a long map phase and its reducers are mostly sitting idle
and filling up slots).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message