hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
Date Sun, 20 Jul 2014 19:11:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068024#comment-14068024

Craig Welch commented on YARN-2008:

Hmm, I don't think that we can change headroom to be just the guaranteed or base capacity
because I believe that will defeat the support for  having a max capacity above the base capacity.
 As I understand it that is in place so that busy queues can grow to use more of the cluster
when other queues are underutilized - to achieve more efficient and full use of the cluster
overall - and if the application gets the low baseline headroom it will not be able to effectively
use that greater capacity.  Assuming we keep support for the max capacity, then even with
pre-emption I think we will need this logic, because preemption won't guarantee that all the
queues have their max capacity available to them, as the total max capacity can be over 100%.
 Preemption will help, certainly, but I don't think it can replace this logic - I think we
need both.

> CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure

> -----------------------------------------------------------------------------------------
>                 Key: YARN-2008
>                 URL: https://issues.apache.org/jira/browse/YARN-2008
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.3.0
>            Reporter: Chen He
>            Assignee: Chen He
>         Attachments: YARN-2008.1.patch, YARN-2008.2.patch
> If there are two queues, both allowed to use 100% of the actual resources in the cluster.
Q1 and Q2 currently use 50% of actual cluster's resources and there is not actual space available.
If we use current method to get headroom, CapacityScheduler thinks there are still available
resources for users in Q1 but they have been used by Q2. 
> If the CapacityScheduelr has a hierarchy queue structure, it may report incorrect queueMaxCap.
Here is a example
>                              ||                    ||rootQueue||     ||
> |  |                               /                               |                
   \                     |
> |      L1ParentQueue1                      |          |            L1ParentQueue2   
> |  (allowed to use up 80% of its parent)    |  |         (allowed to use 20% in minimum
of its parent)|
> |                    /   |     \ |                            |  
> |  L2LeafQueue1 |    L2LeafQueue2 |  |     
> |(50% of its parent) |  (50% of its parent in minimum) |   |
> When we calculate headroom of a user in L2LeafQueue2, current method will think L2LeafQueue2
can use 40% (80%*50%) of actual rootQueue resources. However, without checking L1ParentQueue1,
we are not sure. It is possible that L1ParentQueue2 have used 40% of rootQueue resources right
now. Actually, L2LeafQueue2 can only use 30% (60%*50%). 

This message was sent by Atlassian JIRA

View raw message