hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
Date Sun, 21 Aug 2016 09:05:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429644#comment-15429644

Sunil G commented on YARN-4945:

bq.My assertion is that regardless of what containers are already in the selectedCandidates
list, the intra-queue preemption policy would always need to select more.
yes, I also meant that same. There are chances that intra-queue preemption logic may select
a container whihc is already selected. So we will continue and deduct from intra-queue resourceToObtain
and will continue. I added this point to emphasize that intra queue logic will not do anything
to already selected container.

bq.we may want to consider intra-queue preemption configs for dead zone, natural completion,
Make sense. i will add this point

bq.Is this step calculating the total of preemptable resources for apps in this queue, per
When we consider resource distribution in a queue, there can be resource over subscription
consider the fact that there were no demand at that time when these resource were allocated
to queue/app. Later at a point , few more apps came in and caused resource distribution variation
based on priority or user-limit. In such cases, we will be  considering priority and user-limit
 as separate. 
- priority : all pending resource requests for this app will become “resource to obtain
for this app”
- user-limit: may be partial or full pending resource request will become  “resource to
obtain for this app”. This is depending on “user-limit_headroom - current_used”. This
much can be considered as demand from this app.
I used pending because of the notion from scheduler. But in preemption world, that will be
mapped to resourceTo Obtain. 

And yes, we consider this resourceToObtain per partition level and all calculations are done
as per same.

bq.Is this saying that, when marking containers for preemption, if an app is under its user
limit percent, its containers will not be marked?
I can clarify this. intra-queue preemption will first calculate resourceTOObtain from those
apps which are of high priority (user-limit: those apps which are over-subscribing resource
which crosses its user-limit-quota at given instance). From  these selected apps, we get how
much pending  is there and thus will contribute as resourceToObtain (user-limit: in this case,
we find those apps which are starving and not getting its user-limit-quota).

IN these cases, we will come across apps which is already met / more than its user-limit quota
(for priority). So these apps will be skipped and it will be attribute to resourceToObtain.

bq.Perhaps these should be totally separate policies.
My idea is to come with IntraQueue framework and apply policies like priority and user-limit
on top of that. So with this poc, i am coming with framework and priority preemption. user-limit
can be be added as new policy on top of this framework. And it will be having the points which
are mentioned by you. However for doc, it will be goof if we could have it common for priority
and user-limit. And we can add the point which you have given in comment to doc. This will
give a better insight for intra-queue preemption. Thoughts?

> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch
> This is umbrella ticket to track efforts of preemption within a queue to support features
> YARN-2009. YARN-2113. YARN-4781.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message