hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MENG DING (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Fri, 22 May 2015 21:33:23 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14556868#comment-14556868

MENG DING commented on YARN-1197:

So to summarize the current dilemma:

- A container resource increase request has been granted, and a token has been issued to AM,
- The increase action has not been fulfilled, and the token is not expired yet

- AM can initiate a container resource decrease action to NM, and NM will fulfill it and notify
RM, and then
- Before the toke expires, AM can still initiate a container resource increase action to NM
with the token, and NM will fulfill it and notify RM

Proposed solution:
- When RM receives a container decrease message from NM, it will first check if there is an
outstanding container increase action (by checking the ContainerResourceIncreaseExpirer)
- If the answer is no, RM will go ahead and update its internal resource bookkeeping and reduce
the container resource allocation for this container.
- If the answer is yes, RM will skip the resource reduction in this cycle, keep the resource
decrease message in its newlyDecreasedContainers data structure, and check again in the next
NM-RM heartbeat cycle.
- If in the next heartbeat, a resource increase message to the same container comes, the previous
resource decrease message will be dropped.

Not sure if there are better solution to this problem. Let me know if this makes sense or


> Support changing resources of an allocated container
> ----------------------------------------------------
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>         Attachments: mapreduce-project.patch.ver.1, tools-project.patch.ver.1, yarn-1197-scheduler-v1.pdf,
yarn-1197-v2.pdf, yarn-1197-v3.pdf, yarn-1197-v4.pdf, yarn-1197-v5.pdf, yarn-1197.pdf, yarn-api-protocol.patch.ver.1,
yarn-pb-impl.patch.ver.1, yarn-server-common.patch.ver.1, yarn-server-nodemanager.patch.ver.1,
> The current YARN resource management logic assumes resource allocated to a container
is fixed during the lifetime of it. When users want to change a resource 
> of an allocated container the only way is releasing it and allocating a new container
with expected size.
> Allowing run-time changing resources of an allocated container will give us better control
of resource usage in application side

This message was sent by Atlassian JIRA

View raw message