hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
Date Wed, 16 Nov 2016 01:22:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668964#comment-15668964
] 

Carlo Curino commented on YARN-5881:
------------------------------------

[~seanpo03] thanks for raising this JIRA. This is indeed a very important topic. Beside cloud
settings, where clearly users care to get a fixed amount of resources they paid for and not
a relative amount, this is also important in some on-prem settings, where certain production
queues have fixed jobs running on a schedule that must run (mission critical), and required
fixed amount of resources. In the reservation subsystem as you know we deal with this as 
reservations (dynamic leaf queues), but the same concept should be applied more generally
to the queue structure. This will also be important to support services with gang-semantics.


Beside the general engineering involved, I see a non-trivial issue related of what to do when
capacity fluctuate up/down. I assume you will have buffers to accommodate modest fluctuations,
but what happens if we loose enough capacity to drop below the amount of absolutely configured
queues. You could prioritize certain queues over others, uniformly shrink all queues, etc.

Few questions to answers:
 #  do we allow for a mix of absolutely and relatively configured queues? 
 # how are capacity fluctuations managed?
 # how is "over-capacity" resources distributed? (I can imagine to instantaneously cast both
capacity in the relative domain and perform standard calculations)
 # same as above for preemption actions.
 # can we do this cleanly in CapacityScheduler? (as I mention in other JIRAs the interaction
between many of the tunables is become very unclear)

Overall I think this is very important, and even solving part of the problem under some simplifying
assumption might be ok. 




> Enable configuration of queue capacity in terms of absolute resources
> ---------------------------------------------------------------------
>
>                 Key: YARN-5881
>                 URL: https://issues.apache.org/jira/browse/YARN-5881
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Sean Po
>            Assignee: Sean Po
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a proportion
to cluster capacity. In the context of Yarn being used as a public cloud service, it makes
more sense if queues can be configured absolutely. This will allow administrators to set usage
limits more concretely and simplify customer expectations for cluster allocation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message