hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Tue, 05 Jan 2016 21:55:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083905#comment-15083905
] 

Wangda Tan commented on YARN-1011:
----------------------------------

bq. So, hopefully we can make the policy quite configurable so that the amount of disfavoring
can be tuned for various workloads.
I also +1 to make this policy configurable. This is way I filed YARN-4511.

Instead of threshold of "current" instant usage, I think we can use the threshold of "record"
usage. For example, only 50% of *maximum-under-utilized* resource of past 30 min for each
NM can be used to allocate opportunistic containers. This could reduce number of opportunistic
containers we can allocate, but also this could reduce risks that opportunistic containers
affect other containers.

bq. Can we make this per-resource? (80% memory, 120% CPU)?
+1


> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message