hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2190) Provide a Windows container executor that can limit memory and CPU
Date Wed, 04 Mar 2015 14:17:06 GMT

    [ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346911#comment-14346911
] 

Varun Vasudev commented on YARN-2190:
-------------------------------------

[~chuanliu] one more issue with the patch -
{noformat}
 if (conf.getBoolean(YarnConfiguration.NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED,
+          YarnConfiguration.DEFAULT_NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED)) {
+        int vcores = resource.getVirtualCores();
+        // cap overall usage to the number of cores allocated to YARN
+        float yarnProcessors = NodeManagerHardwareUtils.getContainersCores(
+            ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf),
+            conf);
+        // CPU should be set to a percentage * 100, e.g. 20% cpu rate limit
+        // should be set as 20 * 100. The following setting is equal to:
+        // 100 * (100 * (vcores / Total # of cores allocated to YARN))
+        cpuRate = Math.min(10000, (int) ((vcores * 10000) / yarnProcessors));
+      }
{noformat}

This may not behave as users expected. The 'yarnProcessors' that you receive from the NodeManagerHardwareUtils
is the number of physical cores allocated to YARN containers. However resource.getVirtualCores()
returns a number that the user submits(and can be potentially greater than 'yarnProcessors').
For example, an admin sets 'yarn.nodemanager.resource.cpu-vcores' to 24 on a node with 4 cores(this
can be done by admins who wish to oversubscribe nodes). He also sets 'yarn.nodemanager.resource.percentage-physical-cpu-limit'
to 50 indicating that only 2 physical cores are to be used for YARN containers. The RM allocates
two containers with 12 vcores each on the node. According to your math both containers would
get 100% cpu, when each container should only get 25% cpu.

What you need to do is scale the container vcores to the number of physical cores and not
use the value as provided.


> Provide a Windows container executor that can limit memory and CPU
> ------------------------------------------------------------------
>
>                 Key: YARN-2190
>                 URL: https://issues.apache.org/jira/browse/YARN-2190
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>            Reporter: Chuan Liu
>            Assignee: Chuan Liu
>         Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch,
YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch,
YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch
>
>
> Yarn default container executor on Windows does not set the resource limit on the containers
currently. The memory limit is enforced by a separate monitoring thread. The container implementation
on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory
limits on the job objects. We want to create a Windows container executor that sets the limits
on job objects thus provides resource enforcement at OS level.
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message