hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2190) Provide a Windows container executor that can limit memory and CPU
Date Wed, 17 Sep 2014 20:20:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137903#comment-14137903
] 

Varun Vasudev commented on YARN-2190:
-------------------------------------

[~chuanliu] thanks for the patch! Some questions and comments -
1. What is the behaviour of a process that tries to exceed the allocated memory? Will it start
swapping or will it be killed?
2. Your code assumes a 1-1 mapping of physical cores to vcores. This assumption is/will be
problematic, especially in heterogeneous clusters. You're better off using the ratio of (container-vcores/node-vcores)
to determine cpu limits.
3.
{noformat}
Index: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
===================================================================
--- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
(revision 1618292)
+++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
(working copy)
@@ -38,6 +38,7 @@
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.permission.FsPermission;
 import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.Resource;
 import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
 import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerDiagnosticsUpdateEvent;
@@ -257,6 +258,11 @@
       readLock.unlock();
     }
   }
+
+  protected String[] getRunCommand(String command, String groupId,
+      Configuration conf) {
+    return getRunCommand(command, groupId, conf, null);    
+  }

Index: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
===================================================================
--- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
(revision 1618292)
+++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
(working copy)
@@ -185,7 +185,7 @@
 
       // Setup command to run
       String[] command = getRunCommand(sb.getWrapperScriptPath().toString(),
-        containerIdStr, this.getConf());
+        containerIdStr, this.getConf(), container.getResource());
 
       LOG.info("launchContainer: " + Arrays.toString(command));
{noformat}

Can you explain why you are modifying DefaultContainerExecutor? You've added a method for
the old signature in ContainerExecutor.
4. Can you modify the comments/usage to specify the units of memory(bytes, MB, GB)?

> Provide a Windows container executor that can limit memory and CPU
> ------------------------------------------------------------------
>
>                 Key: YARN-2190
>                 URL: https://issues.apache.org/jira/browse/YARN-2190
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>            Reporter: Chuan Liu
>            Assignee: Chuan Liu
>         Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.2.patch,
YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch
>
>
> Yarn default container executor on Windows does not set the resource limit on the containers
currently. The memory limit is enforced by a separate monitoring thread. The container implementation
on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory
limits on the job objects. We want to create a Windows container executor that sets the limits
on job objects thus provides resource enforcement at OS level.
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message