hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
Date Thu, 08 Feb 2018 18:23:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357353#comment-16357353

Eric Payne commented on YARN-7813:

Thanks @jlowe for the comments!

I'm attaching a new patch that addresses your comments. It does not cherry-pick cleanly to
some of the previous branches. I am working on patches for those other branches.

Will this cause queues to start performing intra-queue preemption that did not previously
with the same configs or vice-versa?

No. Previously, in-queue preemption was enabled and disabled at a particular queue level via
the cross-queue preemption config property. If nothing changes in the configs, this behavior
will remain the same. For example, if cross-queue preemption is disabled at the root queue
and then enabled at root.QueueA, all children of QueueA will have both cross-queue and in-queue
enabled. The intra-queue preemption will only be disabled at a particular level or below if
the new property is set.

bq. "intreQueuePreemptionDisabled" s/b "intraQueuePreemptionDisabled"

Why does CapacitySchedulerLeafQueueInfo have extra logic for getting intra-queue preemption
disabled status? I don't see this similar logic elsewhere in the code.
Yeah, I missed that. The desired behavior outlined above (if configs don't change, intra-queue
enablement doesn't change) was not working quite right, so I moved some of the logic above
the {{getIntraQueuePreemption}} and I didn't make the change where it was really important
({{IntraQueueCandidatesSelector}}. I failed to do my usual rigorous testing so I missed it.
I rectified the problem by adding the {{AbstractCSQueue#getIntraQueuePreemptionDisabledInHierarchy}}
and then putting the cross-queue <-> in-queue dependency logic in {{AbstractCSQueue#getIntraQueuePreemptionDisabled}}.

bq. Technically the queue CLI output changes are incompatible per ...
Yup. I changed the output strings for preemption back to what they were before. I like that
better anyway ;-)

> Capacity Scheduler Intra-queue Preemption should be configurable for each queue
> -------------------------------------------------------------------------------
>                 Key: YARN-7813
>                 URL: https://issues.apache.org/jira/browse/YARN-7813
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacity scheduler, scheduler preemption
>    Affects Versions: 2.9.0, 2.8.3, 3.0.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>            Priority: Major
>         Attachments: YARN-7813.001.patch, YARN-7813.002.patch
> Just as inter-queue (a.k.a. cross-queue) preemption is configurable per queue, intra-queue
(a.k.a. in-queue) preemption should be configurable per queue. If a queue does not have a
setting for intra-queue preemption, it should inherit its parents value.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message