hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.
Date Wed, 07 Mar 2018 09:14:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389289#comment-16389289

Sunil G commented on YARN-4781:

Thanks [~eepayne] for bringing this thread alive..

bq.Then, while all 4 users have running apps, User5 comes along and can't get any resources,
they see that User1 is using 62% more resources than everyone else, and wonders why they can't
get any resources.

Ideally when app3 finishes, quota will go for apps which are started first or less consumed
resources compared to others. However my concern is over case where we need to worry about
priority with FairOrdering. Both are kind of mutually exclusive feature to me.


Coming to the approach, i guess we need to ensure that FairOrdering + USERLIMIT_FIRST is the
combination. In this combination, we will be able to do preemption for apps which under allocated
as per FairOrdering policy.


The patch is simpler enough to cover the comparator change. Thinking out load, could we use
Fair plugin instead of FifoIntraQueuePreemptionPlugin, and make an abstract class to move
all common code. Then Fair plugin could be auto chosen based on ordering policy selected.

> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>                 Key: YARN-4781
>                 URL: https://issues.apache.org/jira/browse/YARN-4781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Major
>         Attachments: YARN-4781.001.patch
> We introduced fairness queue policy since YARN-3319, which will let large applications
make progresses and not starve small applications. However, if a large application takes the
queue’s resources, and containers of the large app has long lifespan, small applications
could still wait for resources for long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to preempt resources
of queue with fairness policy enabled.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message