hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Mon, 22 Feb 2016 16:41:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157248#comment-15157248
] 

Karthik Kambatla commented on YARN-1011:
----------------------------------------

bq. Could you clarify 2.2? What would trigger this if not nodeUpdate()?
We could just periodically go through all the nodes and allocate containers. This is very
similar to continuous/asynchronous scheduling in Fair/Capacity schedulers. 

bq. 2.3. Whats the reasoning behind this? Over-allocating a node seems to be a local decision
based on the nodes expected and actual utilization. So I would expect the logic to be something
similar to 1) Node is already 100% allocated 2) Actual utilization is < 80% 3) Over-allocate
to bring actual utilization ~=80%.
There is a node config that determines if the node allows oversubscription and by how much.
That said, the RM still has to decide when/where to allocate opportunistic containers. When
the overall cluster utilization is low, it is highly likely the RM would find a guaranteed
container soon after it allocates an opportunistic container for a ResourceRequest. By waiting
for this utilization to be over a threshold, we are avoiding having to promote containers
right after allocating them. This shouldn't be a problem in practice, because we expect over-allocation
to help improve the utilization on a fully-allocated cluster. 

{quote}
3. What is the AM/RM interaction in this promotion?
3.2. Not clear what is actually happening here? Will new container be allocated and the opportunistic
container allowed to continue till is exits or is preempted?
{quote}
We don't know yet. :)

Promoting a container on the same node is fairly straight-forward: the node just promotes
the container and the AM can be informed that a running container has been promoted should
it want to differentiate between opportunistic and guaranteed containers. 

I am not actively thinking about promotion across nodes. Given the additional complexity,
I feel we should see some numbers before going further. And, the rest of the work is required
anyway. 

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message