hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Sat, 20 Feb 2016 17:23:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155692#comment-15155692
] 

Karthik Kambatla commented on YARN-1011:
----------------------------------------

Had offline discussions with [~jlowe], [~nroberts], [~elgoiri], [~kkaranasos] and [~asuresh].
Take-aways:
# To ensure the guaranteed containers continue to be allocated exactly the same way as today,
we leave that scheduling logic as is. 
# A "second scheduler" is responsible for allocating opportunistic containers. 
## This "second scheduler" could be another method that is called during node update, or just
another thread that runs asynchronously.
## Using an asynchronous thread allows us to process the nodes in the order of unused resources
instead of node heartbeat.
## Opportunistic scheduling could trigger only after the cluster allocation is over a threshold
- initially, we could hard code it to 80% of cluster capacity. 
# When the scheduler comes around to allocate a guaranteed container for a previously allocated
opportunistic container, that container is promoted.
## Promotion on the same node is straight-forward and always desirable.
## Promotion across nodes is more complicated and leads to resource wastage. However, not
promoting could lead to an application getting resources later than what it would have with
oversubscription turned off. Accordingly, we could have a policy to enable/disable cross-node
promotion. To begin with, it would be disabled. We could always add the option of enabling
it in the future. 

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message