hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Thu, 03 Oct 2013 17:15:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785365#comment-13785365

Bikas Saha commented on YARN-1197:

Alejandro, we have already iterated through the create new container and merge with old container
approach and saw no advantage over a cleaner api that asks for increase in an existing containers
request. Internal implementations of schedulers can choose to model this as a new allocation
followed by an internal merge instead of exposing a new container to the external environment
only to have it merged back by the AM. This is more burden on the AM, NM and confusing to
tracking services like logs/AHS. Also the delta container request would have to specify specific
locality to place it exactly where the existing container is or else the RM can allocate the
delta anywhere. This should just happen automatically if the request was to increase an existing
containers capacity.

Given the above, here is the current approach based on all the comments above

To increase container resource
1) AM ask RM to increase resources
2) RM increases the resources and provides a new token to sign that increase
3) AM calls new NM API to increase the resources, signed with above token

To decrease container resource
1) AM ask NM to decrease the resource
2) NM tell RM about this 
3) RM informs the AM about the change and allocates free space somewhere else

The up side of this is that it avoids race conditions. The down side is that the flow is different
in both cases. Another downside is that the NM does not know if the reduced size is going
to be unsupported by the scheduler. IMO, its not a good idea for RM scheduler internals to
leak into the NM and NM should not get involved in those kind of decisions. One simple solution
would be that when the NM tells the RM about reduced resources, then the RM only reduces its
internal book-keeping by the amount until min-allocation is reached. The extra freed resources
would sit unused in the node but they would anyways be unused if min-allocation was being
enforced by the NM. So we are not worse off. As far as the AM is concerned it already promised
to use stay within the reduced capacity and so this shouldnt affect it.

> Support changing resources of an allocated container
> ----------------------------------------------------
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>         Attachments: yarn-1197.pdf
> Currently, YARN cannot support merge several containers in one node to a big container,
which can make us incrementally ask resources, merge them to a bigger one, and launch our
processes. The user scenario is described in the comments.

This message was sent by Atlassian JIRA

View raw message