hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2056) Disable preemption at Queue level
Date Sun, 21 Sep 2014 01:11:35 GMT

     [ https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Eric Payne updated YARN-2056:
    Attachment: YARN-2056.201409210049.txt

Hi [~leftnoteasy]. Thank you for spending the time to look at this patch and provide helpful

IMHO, the right place to put reserving resource logic for un-preemptable queue is not {{resetCapacity}},
it should in {{computeFixpointAllocation}}.
Does this make sense to you?

Yes, that makes sense, and I think it is a simpler algorithm. I updated the patch, so please
have a look.

I have made a conscious decision to only allow disable preemption at the leaf queue level.
This is because there may be a use case where you want to disable preemption at the parent
level, and have other queue hierarchies leave it alone, but then allow preemption between
children of the disabled parent. So, rather than solve that problem with this fix, I only
allow leaf queues to disable preemption. Even if a leaf queue could inherit it's parent's
disable preemption value, there will likely be cases where part of the parent queue's over-capacity
resources are untouchable and part of them are preemptable.

So, I adjusted your suggested algorithm somewhat. 

- I collected untouchableExtra instead of preemptableExtra at the TempQueue level. in {{computeFixpointAllocation}},
- I looped through each queue, and if one has any untouchableExtra, then the queue's {{idealAssigned
= guaranteed + untouchableExtra}}
- In {{TempQueue#offer}}, one of the calculations is {{current + pending - idealAssigned}}.
I had to take into consideration that if the queue has over capacity, some of it may be untouchable
and some may be preemptable. If some of it is preemptable, then {{current}} could be greater
than {{idealAssigned}}, and {{TempQueue#offer}} would end up assigning more to that queue
than it should.

> Disable preemption at Queue level
> ---------------------------------
>                 Key: YARN-2056
>                 URL: https://issues.apache.org/jira/browse/YARN-2056
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Mayank Bansal
>            Assignee: Eric Payne
>         Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt, YARN-2056.201408310117.txt,
YARN-2056.201409022208.txt, YARN-2056.201409181916.txt, YARN-2056.201409210049.txt
> We need to be able to disable preemption at individual queue level

This message was sent by Atlassian JIRA

View raw message