hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8808) Use aggregate container utilization instead of node utilization to determine resources available for oversubscription
Date Mon, 24 Sep 2018 17:11:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626131#comment-16626131
] 

Haibo Chen commented on YARN-8808:
----------------------------------

Hi [~asuresh]. Not sure if I follow you correctly. Sounds like you are referring to aggregateUtilization
as the aggregate resource *ALLOCATION* of all containers, right?

In the case of Fair Scheduler + oversubscription, aggregateUtilization is the aggregate resource
*UTILIZATION* of all containers on a node (rather than aggregate allocation).

The issue here came up in our testing configuration where only a fraction of the node's hardware
resources is allowed to run containers (Say the node has 10GB memory and 10 vcores, but through
configuration we allow the RM to only see 8GB and 8vcores).  Therefore, I think the scheduler
side should just see numbers based on the configured node capacity (8GB, 8vcores). nodeUtlization
by default, is detected from OS and is therefore based on the node's actually capacity (10GB,
10vcores).

aggregateUtilization/nodeUtilization would tell us how much percentage of the node utilization
is attributed to running containers, no?

> Use aggregate container utilization instead of node utilization to determine resources
available for oversubscription
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-8808
>                 URL: https://issues.apache.org/jira/browse/YARN-8808
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: YARN-1011
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>            Priority: Major
>         Attachments: YARN-8088-YARN-1011.01.patch, YARN-8808-YARN-1011.00.patch
>
>
> Resource oversubscription should be bound to the amount of the resources that can be
allocated to containers, hence the allocation threshold should be with respect to aggregate
container utilization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message