hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
Date Mon, 22 Oct 2018 21:59:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659739#comment-16659739
] 

Haibo Chen commented on YARN-8911:
----------------------------------

Uploaded a new patch that

1) updates the ResourceUtilization cpu javadoc to reduce ambiguity.

2) updates use of ResourceUtilization in YARN-4597

3 ) reverts the core YARN-6870 changes which fixed a floating point inaccuracy issue that
no longer applies.

[~elgoiri] [~asuresh] Would appreciate your reviews. CC [~brookz].

> ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-8911
>                 URL: https://issues.apache.org/jira/browse/YARN-8911
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>            Priority: Major
>         Attachments: YARN-8911.00.patch, YARN-8911.01.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization (ResourceUtilzation.cpu)
is incorrectly documented as a percentage number in [0, 1.0] in the java doc. It is however,
used as the number of vcores used throughout the system.  YARN-4597 is a victim of the 
misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 1.0], i.e.
the percentage of cpu usage across the node.  However, when Containers Monitor tracks the
total aggregate resource utilization of all containers, it adds up the total number of vcores
used by all running containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the aggregated cpu
container utilization being 6.0, but it's supposed to be always between 0 and 1.0)   See
[the code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message