hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5179) Issue of CPU usage of containers
Date Sat, 18 Mar 2017 02:16:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931007#comment-15931007
] 

Miklos Szegedi edited comment on YARN-5179 at 3/18/17 2:16 AM:
---------------------------------------------------------------

[~manirajv06@gmail.com], could you help me, what am I missing?

I ran a test and the numbers look right to me. Here is the code:
{code}
      float cpuUsagePercentPerCore = pTree.getCpuUsagePercent();
      float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore /
              resourceCalculatorPlugin.getNumProcessors();

      // Multiply by 1000 to avoid losing data when converting to int
      int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000
              * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN);
{code}
And here are the values. I have 4 processors, and we use about 3 processors with our job.
This is 75% of total CPU resources, so the first three numbers look right. I chose 8192 vcores
in yarn.nodemanager.resource.cpu-vcores to have a very different number than the 4 cores.
Then we prorate 75% to 8192 vcores, so we get about 6141 number of vcores, which is 6140747
millivcores. That sounds right. What would you expect in this case?
{code}
resourceCalculatorPlugin.getNumProcessors() = 4
cpuUsagePercentPerCore = 299.84116
cpuUsageTotalCoresPercentage = 74.96029
maxVCoresAllottedForContainers = 8192
nodeCpuPercentageForYARN = 100
milliVcoresUsed = 6140747
{code}
Approach #1 above would give us 3000, which is millirealcores. Why would we multiply and then
divide by resourceCalculatorPlugin.getNumVcoresUsed()?
It is an interesting question though what metric is the most useful and simple from the user
point of view. I think I like the 75% the most.


was (Author: miklos.szegedi@cloudera.com):
[~manirajv06@gmail.com], could you help me, what am I missing?

I ran a test and the numbers look right to me. Here is the code:
{code}
      float cpuUsagePercentPerCore = pTree.getCpuUsagePercent();
      float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore /
              resourceCalculatorPlugin.getNumProcessors();

      // Multiply by 1000 to avoid losing data when converting to int
      int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000
              * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN);
{code}
And here are the values. I have 4 processors, and we use about 3 processors with our job.
This is 75% of total CPU resources, so the first three numbers look right. I chose 8192 vcores
in yarn.nodemanager.resource.cpu-vcores to have a very different number than the 4 cores.
Then we prorate 75% to 8192 vcores, so we get about 6141 number of vcores, which is 6140747
millivcores. That sounds right. What would you expect in this case?
{code}
resourceCalculatorPlugin.getNumProcessors() = 4
cpuUsagePercentPerCore = 299.84116
cpuUsageTotalCoresPercentage = 74.96029
maxVCoresAllottedForContainers = 8192
nodeCpuPercentageForYARN = 100
milliVcoresUsed = 6140747
{code}
Approach #1 above would give us 3000, which is millirealcores. Why would we multiply and then
subtract with resourceCalculatorPlugin.getNumVcoresUsed()?
It is an interesting question though what metric is the most useful and simple from the user
point of view. I think I like the 75% the most.

> Issue of CPU usage of containers
> --------------------------------
>
>                 Key: YARN-5179
>                 URL: https://issues.apache.org/jira/browse/YARN-5179
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.0
>         Environment: Both on Windows and Linux
>            Reporter: Zhongkai Mi
>
> // Multiply by 1000 to avoid losing data when converting to int 
>        int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 
>               * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN); 
> This formula will not get right CPU usage based vcore if vcores != physical cores. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message