hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1969) Fair Scheduler: Add policy for Earliest Deadline First
Date Mon, 12 May 2014 15:13:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995151#comment-13995151

Karthik Kambatla commented on YARN-1969:

Just wanted to get a feel for how the policy would shape up before we have an implementation.

A purely EDF approach might not work in clusters with limited resources. Consider two jobs
with the same deadline both requiring the entire cluster. There should be a way to prioritize
between these jobs. Instead of adding a separate policy, I was wondering if it would make
sense to allow FairShare and DRF policies to consider the deadline (if provided) while scheduling
these jobs. A simple way might be to have a deadline-based-weight (estimated remaining job
time / available time until deadline).

> Fair Scheduler: Add policy for Earliest Deadline First
> ------------------------------------------------------
>                 Key: YARN-1969
>                 URL: https://issues.apache.org/jira/browse/YARN-1969
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Maysam Yabandeh
>            Assignee: Maysam Yabandeh
> What we are observing is that some big jobs with many allocated containers are waiting
for a few containers to finish. Under *fair-share scheduling* however they have a low priority
since there are other jobs (usually much smaller, new comers) that are using resources way
below their fair share, hence new released containers are not offered to the big, yet close-to-be-finished
job. Nevertheless, everybody would benefit from an "unfair" scheduling that offers the resource
to the big job since the sooner the big job finishes, the sooner it releases its "many" allocated
resources to be used by other jobs.In other words, what we require is a kind of variation
of *Earliest Deadline First scheduling*, that takes into account the number of already-allocated
resources and estimated time to finish.
> http://en.wikipedia.org/wiki/Earliest_deadline_first_scheduling
> For example, if a job is using MEM GB of memory and is expected to finish in TIME minutes,
the priority in scheduling would be a function p of (MEM, TIME). The expected time to finish
can be estimated by the AppMaster using TaskRuntimeEstimator#estimatedRuntime and be supplied
to RM in the resource request messages. To be less susceptible to the issue of apps gaming
the system, we can have this scheduling limited to *only within a queue*: i.e., adding a EarliestDeadlinePolicy
extends SchedulingPolicy and let the queues to use it by setting the "schedulingPolicy" field.

This message was sent by Atlassian JIRA

View raw message