hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miles Crawford (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.
Date Mon, 21 Mar 2016 19:40:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204957#comment-15204957

Miles Crawford commented on YARN-4781:

This is important for our organization.

Currently, if a job in queue A is using 100% of the cluster resources, and a new job arrives
in queue A, it sometimes cannot even get an application master!  Two jobs in queue A should
share their resources fairly with pre-emption.  It should be feasible to use the same preemption
rules for contention within a queue as for contention across queues.

We have to work around this currently by giving every job a unique name - for example, we
submit jobs to queue "A-2016-03-21T08:23".  Then each job is in its own queue, and can use
preemption to make progress.

> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>                 Key: YARN-4781
>                 URL: https://issues.apache.org/jira/browse/YARN-4781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> We introduced fairness queue policy since YARN-3319, which will let large applications
make progresses and not starve small applications. However, if a large application takes the
queue’s resources, and containers of the large app has long lifespan, small applications
could still wait for resources for long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to preempt resources
of queue with fairness policy enabled.

This message was sent by Atlassian JIRA

View raw message