hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Mon, 15 Jun 2015 23:36:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587146#comment-14587146
] 

Wangda Tan commented on YARN-1197:
----------------------------------

[~sandyr],
Thanks for replying,

bq. Why does the AM need to poll the NM about increase status before taking action? Does the
NM need to do anything other than update its tracking of the resources allotted to the container?
Yes, NM only needs to update tracking of the resource and cgroups. We cannot assume this can
happen immediately, so we cannot put "container increased" to the same RPC. This is same as
startContainer, even if launching a container is fast in most cases, AM needs to poll NM after
invoked startContainer.

bq. Would option (b) ever be able to achieve this kind of latency?
If you consider all now/future optimizations, such as continous-scheduling / scheduler make
decision at same AM-RM heart-beat. (b) needs one more NM-RM heart-beat interval. I agree with
you, it could be hundreds of milli-seconds (a) vs. multi-seconds (b). when the cluster is
idle.

But I'm wondering do we really need add these complexity to AM before we have mature optimizatons
listed above? And also, if the cluster is busier, we cannot expect the delay as well. I tend
to do (b) now since it's simpler to app developer to use this feature, I'm open to add AM->NM
channel if we have YARN scheduler supports fast scheduling better.


> Support changing resources of an allocated container
> ----------------------------------------------------
>
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>         Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, YARN-1197_Design.pdf
>
>
> The current YARN resource management logic assumes resource allocated to a container
is fixed during the lifetime of it. When users want to change a resource 
> of an allocated container the only way is releasing it and allocating a new container
with expected size.
> Allowing run-time changing resources of an allocated container will give us better control
of resource usage in application side



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message