hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MENG DING (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Tue, 16 Jun 2015 15:49:07 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588254#comment-14588254

MENG DING commented on YARN-1197:

Thanks guys for all the comments! I think we all agreed that container decrease request should
go through RM, and decrease action will be triggered with RM->NM heartbeat.

For increase request and action, theoretically option (a) will have better performance. but
we are incurring extra complexity for both YARN and application writers. I was wondering if
we can consider option (c) which sorts of meet (a) and (b) in the middle:

1) AM sends increase request to RM
2) RM allocates the resource and sends the increase token to NM.
3) RM sends response to AM right away, instead of waiting for NM to confirm that the increase
action has been completed.
4) Upon receiving the response (which indicates that the increase has been triggered), AM
should first poll the container status to make sure that the increase is done before taking
action to allocate new tasks.

Option (c) will save one NM-RM heartbeat cycle, and since both option (a) and (c) need to
poll container status, their performance will be very close.

We can have option (b) enabled by default, and use a configuration parameter to turn on option
(c) for framework like Spark.

Do you think if this is worth considering?

> Support changing resources of an allocated container
> ----------------------------------------------------
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>         Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, YARN-1197_Design.pdf
> The current YARN resource management logic assumes resource allocated to a container
is fixed during the lifetime of it. When users want to change a resource 
> of an allocated container the only way is releasing it and allocating a new container
with expected size.
> Allowing run-time changing resources of an allocated container will give us better control
of resource usage in application side

This message was sent by Atlassian JIRA

View raw message