hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Mon, 22 Feb 2016 16:41:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157248#comment-15157248

Karthik Kambatla commented on YARN-1011:

bq. Could you clarify 2.2? What would trigger this if not nodeUpdate()?
We could just periodically go through all the nodes and allocate containers. This is very
similar to continuous/asynchronous scheduling in Fair/Capacity schedulers. 

bq. 2.3. Whats the reasoning behind this? Over-allocating a node seems to be a local decision
based on the nodes expected and actual utilization. So I would expect the logic to be something
similar to 1) Node is already 100% allocated 2) Actual utilization is < 80% 3) Over-allocate
to bring actual utilization ~=80%.
There is a node config that determines if the node allows oversubscription and by how much.
That said, the RM still has to decide when/where to allocate opportunistic containers. When
the overall cluster utilization is low, it is highly likely the RM would find a guaranteed
container soon after it allocates an opportunistic container for a ResourceRequest. By waiting
for this utilization to be over a threshold, we are avoiding having to promote containers
right after allocating them. This shouldn't be a problem in practice, because we expect over-allocation
to help improve the utilization on a fully-allocated cluster. 

3. What is the AM/RM interaction in this promotion?
3.2. Not clear what is actually happening here? Will new container be allocated and the opportunistic
container allowed to continue till is exits or is preempted?
We don't know yet. :)

Promoting a container on the same node is fairly straight-forward: the node just promotes
the container and the AM can be informed that a running container has been promoted should
it want to differentiate between opportunistic and guaranteed containers. 

I am not actively thinking about promotion across nodes. Given the additional complexity,
I feel we should see some numbers before going further. And, the rest of the work is required

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, yarn-1011-design-v2.pdf
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.

This message was sent by Atlassian JIRA

View raw message