hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
Date Tue, 23 Oct 2018 17:12:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660988#comment-16660988

Haibo Chen commented on YARN-8911:

{quote} Is there any test that would track this from the container to the RM?
I have not been able to find any.

With that said, the change included in this patch has nothing to do with the code path that
collects utilization of containers and sends to RM.  Hence, such tests won't cover the changes

The containers and node utilization are still collected by containers monitor and NodeResourceMonitor
,and sent to RM in the heartbeat. What's changed in containers monitor is a helper methods
that gets called by Container Scheduler only, which tracks containers utilization internally.
There are some test coverage on the container utilization maintained by Container Scheduler

> ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
> ------------------------------------------------------------------------------------
>                 Key: YARN-8911
>                 URL: https://issues.apache.org/jira/browse/YARN-8911
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>            Priority: Major
>         Attachments: YARN-8911.00.patch, YARN-8911.01.patch
> *UPDATE*:  *per discussion below, the cpu resource utlization (ResourceUtilzation.cpu)
is incorrectly documented as a percentage number in [0, 1.0] in the java doc. It is however,
used as the number of vcores used throughout the system.  YARN-4597 is a victim of the 
misleading java doc.*
> ResourceUtilization represents the cpu utilization with a float number in [0, 1.0], i.e.
the percentage of cpu usage across the node.  However, when Containers Monitor tracks the
total aggregate resource utilization of all containers, it adds up the total number of vcores
used by all running containers.
> (If you have 6 containers running, each consuming 1 vcore, you'd see the aggregated cpu
container utilization being 6.0, but it's supposed to be always between 0 and 1.0)   See
[the code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message