hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Tue, 16 Jun 2015 00:19:07 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587196#comment-14587196
] 

Wangda Tan commented on YARN-1197:
----------------------------------

bq. To clarify: with proper tuning, we can currently get low hundreds of milliseconds without
adding any new scheduler features. With the new scheduler feature I'm imagining, we'd only
be limited by the RPC + scheduler time, so we could get 10s of milliseconds with proper tuning.
I think this assumes cluster is quite idle, I understand the low latency could be achieved,
but it's not guaranteed since we don't support oversubscribing, etc. If you assume the cluster
is very idle, one solution might be holding more resource at the beginning instead of increasing.
In real environment, I think the expectation of delay should still be seconds level.

>From YARN's perspective, (b) handles most of logic within YARN daemons (instead of AM),
we don't need to consider inconsistency status between RM/AM when doing recovery, that is
really what I prefer :). I'm not against of doing (a), but I prefer to do that when we have
solid foundation for fast scheduling. I'm not sure if there's any resource management platform
in production supports that, but some research papers such as Sparrow uses quite different
protocol/approach than YARN. I expect there're still some TODO items for YARN to get guaranteed
fast scheduling.

> Support changing resources of an allocated container
> ----------------------------------------------------
>
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>         Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, YARN-1197_Design.pdf
>
>
> The current YARN resource management logic assumes resource allocated to a container
is fixed during the lifetime of it. When users want to change a resource 
> of an allocated container the only way is releasing it and allocating a new container
with expected size.
> Allowing run-time changing resources of an allocated container will give us better control
of resource usage in application side



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message