hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Tue, 16 Jun 2015 18:04:08 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588478#comment-14588478
] 

Sandy Ryza commented on YARN-1197:
----------------------------------

bq. I think this assumes cluster is quite idle, I understand the low latency could be achieved,
but it's not guaranteed since we don't support oversubscribing, etc.
If the cluster is fully contended we certainly won't get this performance.  But as long as
there is a decent chunk of space, which is common in many settings, we can.  The cluster doesn't
need to be fully idle by any means.

More broadly, just because YARN is not good at hitting sub-second latencies doesn't mean that
it isn't a design goal.  I strongly oppose any argument that uses the current slowness of
YARN as a justification for why we should make architectural decisions that could compromise
latencies.

That said, I still don't have a strong grasp on the kind of complexity we're introducing in
the AM, so would like to try to understand that before arguing against you further.

Is the main problem we're grappling still the one Meng brought up here:
https://issues.apache.org/jira/browse/YARN-1197?focusedCommentId=14556803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14556803?
I.e. that an AM can receive an increase from the RM, then issue a decrease to the NM, and
then use its increase to get resources it doesn't deserve?

Or is the idea that, even if we didn't have this JIRA, NMClient is too complicated, and we'd
like to reduce that?

> Support changing resources of an allocated container
> ----------------------------------------------------
>
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>         Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, YARN-1197_Design.pdf
>
>
> The current YARN resource management logic assumes resource allocated to a container
is fixed during the lifetime of it. When users want to change a resource 
> of an allocated container the only way is releasing it and allocating a new container
with expected size.
> Allowing run-time changing resources of an allocated container will give us better control
of resource usage in application side



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message