hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Sun, 27 Dec 2015 19:34:50 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15072223#comment-15072223

Karthik Kambatla commented on YARN-1011:

bq. Would it make sense to make over-subscription a global property set by the RM instead
of per-node?
Good question. I thought about it quite some. Here is my reasoning for doing on the NM side.
We can always switch back to defining it to the RM if that makes more sense.
# Even if we have the knob on the RM, the node still has to support it: monitor the resource
usage on the node and kill the OPPORTUNISTIC containers if need be. On a cluster with NMs
of different versions (say, during a rolling upgrade), the RM will have to keep track of NMs
that support over-subscription. So, we do need some config for the NM anyway. Further, there
could be node-specific conditions - hardware, other services running on the node etc. - that
could affect the over-subscription capacity of the node. For instance, it might be okay to
sign up for 90% of the advertised capacity on node A, but only 80% on the node B. And, this
ability to soak up extra work could change over time. 
# In terms of implementation, the node already sends its capacity and its aggregate-container-utilization.
It might as well send an oversubscription-percentage over, which is interpreted as the fraction
of its advertised capacity. e.g. A node with 64 GB memory could advertise its capacity as
50 GB and oversubscription-percentage 0.9. The RM could schedule upto 45 GB of utilization.
An oversubscription-percentage <= 0 would indicate the feature is turned off. 

bq. What would be the first policy to implement? I guess we can define it in YARN-1015.
The simplest policy would likely be just assuming there are more resources on the node, and
continue allocating with the same policies we use today for free/unallocated resources. 
This should work okay for the FairScheduler. I am less familiar with the intricate details
of CS, but would think it should apply there as well. [~leftnoteasy] - thoughts? 

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.

This message was sent by Atlassian JIRA

View raw message