hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From templ...@apache.org
Subject hadoop git commit: YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path (Contributed by Miklos Szegedi via Daniel Templeton)
Date Tue, 08 Aug 2017 17:34:33 GMT
Repository: hadoop
Updated Branches:
  refs/heads/trunk 98912950b -> 47b145b9b


YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
(Contributed by Miklos Szegedi via Daniel Templeton)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/47b145b9
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/47b145b9
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/47b145b9

Branch: refs/heads/trunk
Commit: 47b145b9b4e81d781891abce8a6638f0b436acc4
Parents: 9891295
Author: Daniel Templeton <templedf@apache.org>
Authored: Tue Aug 8 10:33:26 2017 -0700
Committer: Daniel Templeton <templedf@apache.org>
Committed: Tue Aug 8 10:33:26 2017 -0700

----------------------------------------------------------------------
 .../src/main/resources/yarn-default.xml         | 43 ++++++++++-----
 .../linux/resources/CGroupsHandler.java         | 15 +++++
 .../linux/resources/CGroupsHandlerImpl.java     | 26 +++++----
 .../linux/resources/ResourceHandlerModule.java  | 58 ++++++++++++++++++--
 .../util/CgroupsLCEResourcesHandler.java        | 53 ++++++++++++------
 .../linux/resources/TestCGroupsHandlerImpl.java | 27 ++++++++-
 .../util/TestCgroupsLCEResourcesHandler.java    | 31 +++++++++++
 .../src/site/markdown/GracefulDecommission.md   | 12 ++--
 .../src/site/markdown/NodeManagerCgroups.md     | 17 +++++-
 .../site/markdown/WritingYarnApplications.md    |  4 +-
 .../src/site/markdown/registry/yarn-registry.md | 14 ++---
 11 files changed, 237 insertions(+), 63 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 95b8a88..000e892 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -134,7 +134,7 @@
 
   <property>
       <description>
-        This configures the HTTP endpoint for Yarn Daemons.The following
+        This configures the HTTP endpoint for YARN Daemons.The following
         values are supported:
         - HTTP_ONLY : Service is provided only on http
         - HTTPS_ONLY : Service is provided only on https
@@ -1063,14 +1063,14 @@
       DeletionService will delete the application's localized file directory
       and log directory.
       
-      To diagnose Yarn application problems, set this property's value large
+      To diagnose YARN application problems, set this property's value large
       enough (for example, to 600 = 10 minutes) to permit examination of these
       directories. After changing the property's value, you must restart the 
       nodemanager in order for it to have an effect.
 
-      The roots of Yarn applications' work directories is configurable with
+      The roots of YARN applications' work directories is configurable with
       the yarn.nodemanager.local-dirs property (see below), and the roots
-      of the Yarn applications' log directories is configurable with the 
+      of the YARN applications' log directories is configurable with the
       yarn.nodemanager.log-dirs property (see also below).
     </description>
     <name>yarn.nodemanager.delete.debug-delay-sec</name>
@@ -1510,28 +1510,45 @@
   <property>
     <description>The cgroups hierarchy under which to place YARN proccesses (cannot
contain commas).
     If yarn.nodemanager.linux-container-executor.cgroups.mount is false
-    (that is, if cgroups have been pre-configured) and the Yarn user has write
+    (that is, if cgroups have been pre-configured) and the YARN user has write
     access to the parent directory, then the directory will be created.
-    If the directory already exists, the administrator has to give Yarn
+    If the directory already exists, the administrator has to give YARN
     write permissions to it recursively.
-    Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.</description>
+    This property only applies when the LCE resources handler is set to
+    CgroupsLCEResourcesHandler.</description>
     <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
     <value>/hadoop-yarn</value>
   </property>
 
   <property>
     <description>Whether the LCE should attempt to mount cgroups if not found.
-    Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.</description>
+    This property only applies when the LCE resources handler is set to
+    CgroupsLCEResourcesHandler.
+    </description>
     <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
     <value>false</value>
   </property>
 
   <property>
-    <description>Where the LCE should attempt to mount cgroups if not found. Common
locations
-    include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux
-    distribution in use. This path must exist before the NodeManager is launched.
-    Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and
-    yarn.nodemanager.linux-container-executor.cgroups.mount is true.</description>
+    <description>This property sets the path from which YARN will read the
+    CGroups configuration. YARN has built-in functionality to discover the
+    system CGroup mount paths, so use this property only if YARN's automatic
+    mount path discovery does not work.
+
+    The path specified by this property must exist before the NodeManager is
+    launched.
+    If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true,
+    YARN will first try to mount the CGroups at the specified path before
+    reading them.
+    If yarn.nodemanager.linux-container-executor.cgroups.mount is set to
+    false, YARN will read the CGroups at the specified path.
+    If this property is empty, YARN tries to detect the CGroups location.
+
+    Please refer to NodeManagerCgroups.html in the documentation for further
+    details.
+    This property only applies when the LCE resources handler is set to
+    CgroupsLCEResourcesHandler.
+    </description>
     <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
   </property>
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
index 8fc35a8..82bd366 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
@@ -23,6 +23,9 @@ package org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resourc
 import org.apache.hadoop.classification.InterfaceAudience;
 import org.apache.hadoop.classification.InterfaceStability;
 
+import java.util.HashSet;
+import java.util.Set;
+
 /**
  * Provides CGroups functionality. Implementations are expected to be
  * thread-safe
@@ -54,6 +57,18 @@ public interface CGroupsHandler {
     String getName() {
       return name;
     }
+
+    /**
+     * Get the list of valid cgroup names.
+     * @return The set of cgroup name strings
+     */
+    public static Set<String> getValidCGroups() {
+      HashSet<String> validCgroups = new HashSet<>();
+      for (CGroupController controller : CGroupController.values()) {
+        validCgroups.add(controller.getName());
+      }
+      return validCgroups;
+    }
   }
 
   String CGROUP_FILE_TASKS = "tasks";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
index 85b01cd..9fd20eb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
@@ -83,7 +83,7 @@ class CGroupsHandlerImpl implements CGroupsHandler {
    * @param mtab mount file location
    * @throws ResourceHandlerException if initialization failed
    */
-  public CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
+  CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
       privilegedOperationExecutor, String mtab)
       throws ResourceHandlerException {
     this.cGroupPrefix = conf.get(YarnConfiguration.
@@ -115,7 +115,7 @@ class CGroupsHandlerImpl implements CGroupsHandler {
    *                                    PrivilegedContainerOperations
    * @throws ResourceHandlerException if initialization failed
    */
-  public CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
+  CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
       privilegedOperationExecutor) throws ResourceHandlerException {
     this(conf, privilegedOperationExecutor, MTAB_FILE);
   }
@@ -142,11 +142,18 @@ class CGroupsHandlerImpl implements CGroupsHandler {
     // the same hierarchy will be mounted at each mount point with the same
     // subsystem set.
 
-    Map<String, Set<String>> newMtab;
+    Map<String, Set<String>> newMtab = null;
     Map<CGroupController, String> cPaths;
     try {
-      // parse mtab
-      newMtab = parseMtab(mtabFile);
+      if (this.cGroupMountPath != null && !this.enableCGroupMount) {
+        newMtab = ResourceHandlerModule.
+            parseConfiguredCGroupPath(this.cGroupMountPath);
+      }
+
+      if (newMtab == null) {
+        // parse mtab
+        newMtab = parseMtab(mtabFile);
+      }
 
       // find cgroup controller paths
       cPaths = initializeControllerPathsFromMtab(newMtab);
@@ -203,10 +210,8 @@ class CGroupsHandlerImpl implements CGroupsHandler {
       throws IOException {
     Map<String, Set<String>> ret = new HashMap<>();
     BufferedReader in = null;
-    HashSet<String> validCgroups = new HashSet<>();
-    for (CGroupController controller : CGroupController.values()) {
-      validCgroups.add(controller.getName());
-    }
+    Set<String> validCgroups =
+        CGroupsHandler.CGroupController.getValidCGroups();
 
     try {
       FileInputStream fis = new FileInputStream(new File(mtab));
@@ -487,7 +492,8 @@ class CGroupsHandlerImpl implements CGroupsHandler {
       try (BufferedReader inl =
           new BufferedReader(new InputStreamReader(new FileInputStream(cgf
               + "/tasks"), "UTF-8"))) {
-        if ((str = inl.readLine()) != null) {
+        str = inl.readLine();
+        if (str != null) {
           LOG.debug("First line in cgroup tasks file: " + cgf + " " + str);
         }
       } catch (IOException e) {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
index 7fc04bd..4d137f0 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
@@ -31,6 +31,13 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileg
 import org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler;
 import org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler;
 
+import java.io.File;
+import java.io.IOException;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Arrays;
 import java.util.ArrayList;
 import java.util.List;
 
@@ -113,8 +120,8 @@ public class ResourceHandlerModule {
   }
 
   private static TrafficControlBandwidthHandlerImpl
-    getTrafficControlBandwidthHandler(Configuration conf)
-      throws ResourceHandlerException {
+      getTrafficControlBandwidthHandler(Configuration conf)
+        throws ResourceHandlerException {
     if (conf.getBoolean(YarnConfiguration.NM_NETWORK_RESOURCE_ENABLED,
         YarnConfiguration.DEFAULT_NM_NETWORK_RESOURCE_ENABLED)) {
       if (trafficControlBandwidthHandler == null) {
@@ -137,8 +144,8 @@ public class ResourceHandlerModule {
   }
 
   public static OutboundBandwidthResourceHandler
-    getOutboundBandwidthResourceHandler(Configuration conf)
-      throws ResourceHandlerException {
+      getOutboundBandwidthResourceHandler(Configuration conf)
+        throws ResourceHandlerException {
     return getTrafficControlBandwidthHandler(conf);
   }
 
@@ -176,7 +183,7 @@ public class ResourceHandlerModule {
   }
 
   private static CGroupsMemoryResourceHandlerImpl
-    getCgroupsMemoryResourceHandler(
+      getCgroupsMemoryResourceHandler(
       Configuration conf) throws ResourceHandlerException {
     if (cGroupsMemoryResourceHandler == null) {
       synchronized (MemoryResourceHandler.class) {
@@ -229,4 +236,45 @@ public class ResourceHandlerModule {
   static void nullifyResourceHandlerChain() throws ResourceHandlerException {
     resourceHandlerChain = null;
   }
+
+  /**
+   * If a cgroup mount directory is specified, it returns cgroup directories
+   * with valid names.
+   * The requirement is that each hierarchy has to be named with the comma
+   * separated names of subsystems supported.
+   * For example: /sys/fs/cgroup/cpu,cpuacct
+   * @param cgroupMountPath Root cgroup mount path (/sys/fs/cgroup in the
+   *                        example above)
+   * @return A path to cgroup subsystem set mapping in the same format as
+   *         {@link CGroupsHandlerImpl#parseMtab(String)}
+   * @throws IOException if the specified directory cannot be listed
+   */
+  public static Map<String, Set<String>> parseConfiguredCGroupPath(
+      String cgroupMountPath) throws IOException {
+    File cgroupDir = new File(cgroupMountPath);
+    File[] list = cgroupDir.listFiles();
+    if (list == null) {
+      throw new IOException("Empty cgroup mount directory specified: " +
+          cgroupMountPath);
+    }
+
+    Map<String, Set<String>> pathSubsystemMappings = new HashMap<>();
+    Set<String> validCGroups =
+        CGroupsHandler.CGroupController.getValidCGroups();
+    for (File candidate: list) {
+      Set<String> cgroupList =
+          new HashSet<>(Arrays.asList(candidate.getName().split(",")));
+      // Collect the valid subsystem names
+      cgroupList.retainAll(validCGroups);
+      if (!cgroupList.isEmpty()) {
+        if (candidate.isDirectory() && candidate.canWrite()) {
+          pathSubsystemMappings.put(candidate.getAbsolutePath(), cgroupList);
+        } else {
+          LOG.warn("The following cgroup is not a directory or it is not"
+              + " writable" + candidate.getAbsolutePath());
+        }
+      }
+    }
+    return pathSubsystemMappings;
+  }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
index bca4fdc..7a89285 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
@@ -27,6 +27,7 @@ import java.io.InputStreamReader;
 import java.io.OutputStreamWriter;
 import java.io.PrintWriter;
 import java.io.Writer;
+import java.util.Arrays;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashMap;
@@ -39,7 +40,6 @@ import java.util.regex.Pattern;
 
 import com.google.common.annotations.VisibleForTesting;
 
-import com.google.common.collect.Sets;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.apache.hadoop.conf.Configuration;
@@ -51,6 +51,8 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor;
 import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation;
 import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsCpuResourceHandlerImpl;
+import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandler;
+import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerModule;
 import org.apache.hadoop.yarn.util.Clock;
 import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin;
 import org.apache.hadoop.yarn.util.SystemClock;
@@ -87,11 +89,11 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
 
   private long deleteCgroupTimeout;
   private long deleteCgroupDelay;
-  // package private for testing purposes
+  @VisibleForTesting
   Clock clock;
 
   private float yarnProcessors;
-  int nodeVCores;
+  private int nodeVCores;
 
   public CgroupsLCEResourcesHandler() {
     this.controllerPaths = new HashMap<String, String>();
@@ -132,8 +134,10 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
     this.strictResourceUsageMode =
         conf
           .getBoolean(
-            YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
-            YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
+            YarnConfiguration
+                .NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
+            YarnConfiguration
+                .DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
 
     int len = cgroupPrefix.length();
     if (cgroupPrefix.charAt(len - 1) == '/') {
@@ -169,8 +173,10 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
     if (systemProcessors != (int) yarnProcessors) {
       LOG.info("YARN containers restricted to " + yarnProcessors + " cores");
       int[] limits = getOverallLimits(yarnProcessors);
-      updateCgroup(CONTROLLER_CPU, "", CPU_PERIOD_US, String.valueOf(limits[0]));
-      updateCgroup(CONTROLLER_CPU, "", CPU_QUOTA_US, String.valueOf(limits[1]));
+      updateCgroup(CONTROLLER_CPU, "", CPU_PERIOD_US,
+          String.valueOf(limits[0]));
+      updateCgroup(CONTROLLER_CPU, "", CPU_QUOTA_US,
+          String.valueOf(limits[1]));
     } else if (CGroupsCpuResourceHandlerImpl.cpuLimitsExist(
         pathForCgroup(CONTROLLER_CPU, ""))) {
       LOG.info("Removing CPU constraints for YARN containers.");
@@ -178,8 +184,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
     }
   }
 
-  int[] getOverallLimits(float yarnProcessors) {
-    return CGroupsCpuResourceHandlerImpl.getOverallLimits(yarnProcessors);
+  int[] getOverallLimits(float yarnProcessorsArg) {
+    return CGroupsCpuResourceHandlerImpl.getOverallLimits(yarnProcessorsArg);
   }
 
 
@@ -204,7 +210,7 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
       LOG.debug("createCgroup: " + path);
     }
 
-    if (! new File(path).mkdir()) {
+    if (!new File(path).mkdir()) {
       throw new IOException("Failed to create cgroup at " + path);
     }
   }
@@ -251,7 +257,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
       try (BufferedReader inl =
             new BufferedReader(new InputStreamReader(new FileInputStream(cgf
               + "/tasks"), "UTF-8"))) {
-        if ((str = inl.readLine()) != null) {
+        str = inl.readLine();
+        if (str != null) {
           LOG.debug("First line in cgroup tasks file: " + cgf + " " + str);
         }
       } catch (IOException e) {
@@ -337,9 +344,9 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
               (containerVCores * yarnProcessors) / (float) nodeVCores;
           int[] limits = getOverallLimits(containerCPU);
           updateCgroup(CONTROLLER_CPU, containerName, CPU_PERIOD_US,
-            String.valueOf(limits[0]));
+              String.valueOf(limits[0]));
           updateCgroup(CONTROLLER_CPU, containerName, CPU_QUOTA_US,
-            String.valueOf(limits[1]));
+              String.valueOf(limits[1]));
         }
       }
     }
@@ -400,6 +407,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
   private Map<String, Set<String>> parseMtab() throws IOException {
     Map<String, Set<String>> ret = new HashMap<String, Set<String>>();
     BufferedReader in = null;
+    Set<String> validCgroups =
+        CGroupsHandler.CGroupController.getValidCGroups();
 
     try {
       FileInputStream fis = new FileInputStream(new File(getMtabFileName()));
@@ -415,8 +424,11 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
           String options = m.group(3);
 
           if (type.equals(CGROUPS_FSTYPE)) {
-            HashSet<String> value = Sets.newHashSet(options.split(","));
-            ret.put(path, value);
+            Set<String> cgroupList =
+                new HashSet<>(Arrays.asList(options.split(",")));
+            // Collect the valid subsystem names
+            cgroupList.retainAll(validCgroups);
+            ret.put(path, cgroupList);
           }
         }
       }
@@ -448,7 +460,16 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler
{
 
   private void initializeControllerPaths() throws IOException {
     String controllerPath;
-    Map<String, Set<String>> parsedMtab = parseMtab();
+    Map<String, Set<String>> parsedMtab = null;
+
+    if (this.cgroupMountPath != null && !this.cgroupMount) {
+      parsedMtab = ResourceHandlerModule.
+          parseConfiguredCGroupPath(this.cgroupMountPath);
+    }
+
+    if (parsedMtab == null) {
+      parsedMtab = parseMtab();
+    }
 
     // CPU
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
index 7a4d39f..ab989cf 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
@@ -573,4 +573,29 @@ public class TestCGroupsHandlerImpl {
         new File(new File(newMountPoint, "cpu"), this.hierarchy);
     assertTrue("Yarn cgroup should exist", hierarchyFile.exists());
   }
-}
+
+
+  @Test
+  public void testManualCgroupSetting() throws ResourceHandlerException {
+    YarnConfiguration conf = new YarnConfiguration();
+    conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH, tmpPath);
+    conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_HIERARCHY,
+        "/hadoop-yarn");
+    File cpu = new File(new File(tmpPath, "cpuacct,cpu"), "/hadoop-yarn");
+
+    try {
+      Assert.assertTrue("temp dir should be created", cpu.mkdirs());
+
+      CGroupsHandlerImpl cGroupsHandler = new CGroupsHandlerImpl(conf, null);
+      cGroupsHandler.initializeCGroupController(
+              CGroupsHandler.CGroupController.CPU);
+
+      Assert.assertEquals("CPU CGRoup path was not set", cpu.getAbsolutePath(),
+              new File(cGroupsHandler.getPathForCGroup(
+                  CGroupsHandler.CGroupController.CPU, "")).getAbsolutePath());
+
+    } finally {
+      FileUtils.deleteQuietly(cpu);
+    }
+  }
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
index 1ed8fd8..7d8704f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
@@ -41,6 +41,8 @@ import java.util.Scanner;
 import java.util.Set;
 import java.util.concurrent.CountDownLatch;
 
+import static org.mockito.Mockito.when;
+
 @Deprecated
 public class TestCgroupsLCEResourcesHandler {
   private static File cgroupDir = null;
@@ -388,4 +390,33 @@ public class TestCgroupsLCEResourcesHandler {
       FileUtils.deleteQuietly(memory);
     }
   }
+
+  @Test
+  public void testManualCgroupSetting() throws IOException {
+    CgroupsLCEResourcesHandler handler = new CgroupsLCEResourcesHandler();
+    YarnConfiguration conf = new YarnConfiguration();
+    conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH,
+        cgroupDir.getAbsolutePath());
+    handler.setConf(conf);
+    File cpu = new File(new File(cgroupDir, "cpuacct,cpu"), "/hadoop-yarn");
+
+    try {
+      Assert.assertTrue("temp dir should be created", cpu.mkdirs());
+
+      final int numProcessors = 4;
+      ResourceCalculatorPlugin plugin =
+              Mockito.mock(ResourceCalculatorPlugin.class);
+      Mockito.doReturn(numProcessors).when(plugin).getNumProcessors();
+      Mockito.doReturn(numProcessors).when(plugin).getNumCores();
+      when(plugin.getNumProcessors()).thenReturn(8);
+      handler.init(null, plugin);
+
+      Assert.assertEquals("CPU CGRoup path was not set", cpu.getParent(),
+          handler.getControllerPaths().get("cpu"));
+
+    } finally {
+      FileUtils.deleteQuietly(cpu);
+    }
+  }
+
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md
index 2acb3d2..2e83ca2 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md
@@ -13,7 +13,7 @@
 -->
 
 
-Graceful Decommission of Yarn Nodes
+Graceful Decommission of YARN Nodes
 ===============
 
 * [Overview](#overview)
@@ -29,19 +29,19 @@ Graceful Decommission of Yarn Nodes
 Overview
 --------
 
-Yarn is scalable very easily: any new NodeManager could join to the configured ResourceManager
and start to execute jobs. But to achieve full elasticity we need a decommissioning process
which helps to remove existing nodes and down-scale the cluster.
+YARN is scalable very easily: any new NodeManager could join to the configured ResourceManager
and start to execute jobs. But to achieve full elasticity we need a decommissioning process
which helps to remove existing nodes and down-scale the cluster.
 
-Yarn Nodes could be decommissioned NORMAL or GRACEFUL.
+YARN Nodes could be decommissioned NORMAL or GRACEFUL.
 
-Normal Decommission of Yarn Nodes means an immediate shutdown.
+Normal Decommission of YARN Nodes means an immediate shutdown.
 
-Graceful Decommission of Yarn Nodes is the mechanism to decommission NMs while minimize the
impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule
new containers on it and will wait for running containers and applications to complete (or
until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED.
+Graceful Decommission of YARN Nodes is the mechanism to decommission NMs while minimize the
impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule
new containers on it and will wait for running containers and applications to complete (or
until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED.
 
 ## Quick start
 
 To do a normal decommissioning:
 
-1. Start a Yarn cluster (with NodeManageres and ResourceManager)
+1. Start a YARN cluster (with NodeManageres and ResourceManager)
 2. Start a yarn job (for example with `yarn jar...` )
 3. Add `yarn.resourcemanager.nodes.exclude-path` property to your `yarn-site.xml` (Note:
you don't need to restart the ResourceManager)
 4. Create a text file (the location is defined in the previous step) with one line which
contains the name of a selected NodeManager 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md
index 2704f10..d362801 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md
@@ -17,7 +17,7 @@ Using CGroups with YARN
 
 <!-- MACRO{toc|fromDepth=0|toDepth=3} -->
 
-CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children,
into hierarchical groups with specialized behaviour. CGroups is a Linux kernel feature and
was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to
be limited in their resource usage. A good example of this is CPU usage. Without CGroups,
it becomes hard to limit container CPU usage. Currently, CGroups is only used for limiting
CPU usage.
+CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children,
into hierarchical groups with specialized behaviour. CGroups is a Linux kernel feature and
was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to
be limited in their resource usage. A good example of this is CPU usage. Without CGroups,
it becomes hard to limit container CPU usage.
 
 CGroups Configuration
 ---------------------
@@ -30,9 +30,9 @@ The following settings are related to setting up CGroups. These need to
be set i
 |:---- |:---- |
 | `yarn.nodemanager.container-executor.class` | This should be set to "org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor".
CGroups is a Linux kernel feature and is exposed via the LinuxContainerExecutor. |
 | `yarn.nodemanager.linux-container-executor.resources-handler.class` | This should be set
to "org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler". Using the
LinuxContainerExecutor doesn't force you to use CGroups. If you wish to use CGroups, the resource-handler-class
must be set to CGroupsLCEResourceHandler. |
-| `yarn.nodemanager.linux-container-executor.cgroups.hierarchy` | The cgroups hierarchy under
which to place YARN proccesses(cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount
is false (that is, if cgroups have been pre-configured) and the Yarn user has write access
to the parent directory, then the directory will be created. If the directory already exists,
the administrator has to give Yarn write permissions to it recursively. |
+| `yarn.nodemanager.linux-container-executor.cgroups.hierarchy` | The cgroups hierarchy under
which to place YARN proccesses(cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount
is false (that is, if cgroups have been pre-configured) and the YARN user has write access
to the parent directory, then the directory will be created. If the directory already exists,
the administrator has to give YARN write permissions to it recursively. |
 | `yarn.nodemanager.linux-container-executor.cgroups.mount` | Whether the LCE should attempt
to mount cgroups if not found - can be true or false. |
-| `yarn.nodemanager.linux-container-executor.cgroups.mount-path` | Where the LCE should attempt
to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default
location can vary depending on the Linux distribution in use. This path must exist before
the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler,
and yarn.nodemanager.linux-container-executor.cgroups.mount is true. A point to note here
is that the container-executor binary will try to mount the path specified + "/" + the subsystem.
In our case, since we are trying to limit CPU the binary tries to mount the path specified
+ "/cpu" and that's the path it expects to exist. |
+| `yarn.nodemanager.linux-container-executor.cgroups.mount-path` | Optional. Where CGroups
are located. LCE will try to mount them here, if `yarn.nodemanager.linux-container-executor.cgroups.mount`
is true. LCE will try to use CGroups from this location, if `yarn.nodemanager.linux-container-executor.cgroups.mount`
is false. If specified, this path and its subdirectories (CGroup hierarchies) must exist and
they should be readable and writable by YARN before the NodeManager is launched. See CGroups
mount options below for details. |
 | `yarn.nodemanager.linux-container-executor.group` | The Unix group of the NodeManager.
It should match the setting in "container-executor.cfg". This configuration is required for
validating the secure access of the container-executor binary. |
 
 The following settings are related to limiting resource usage of YARN containers:
@@ -42,6 +42,17 @@ The following settings are related to limiting resource usage of YARN containers
 | `yarn.nodemanager.resource.percentage-physical-cpu-limit` | This setting lets you limit
the cpu usage of all YARN containers. It sets a hard upper limit on the cumulative CPU usage
of the containers. For example, if set to 60, the combined CPU usage of all YARN containers
will not exceed 60%. |
 | `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage` | CGroups allows
cpu usage limits to be hard or soft. When this setting is true, containers cannot use more
CPU usage than allocated even if spare CPU is available. This ensures that containers can
only use CPU that they were allocated. When set to false, containers can use spare CPU if
available. It should be noted that irrespective of whether set to true or false, at no time
can the combined CPU usage of all containers exceed the value specified in "yarn.nodemanager.resource.percentage-physical-cpu-limit".
|
 
+CGroups mount options
+---------------------
+
+YARN uses CGroups through a directory structure mounted into the file system by the kernel.
There are three options to attach to CGroups.
+
+| Option | Description |
+|:---- |:---- |
+| Discover CGroups mounted already | This should be used on newer systems like RHEL7 or Ubuntu16
or if the administrator mounts CGroups before YARN starts. Set `yarn.nodemanager.linux-container-executor.cgroups.mount`
to false and leave other settings set to their defaults. YARN will locate the mount points
in `/proc/mounts`. Common locations include `/sys/fs/cgroup` and `/cgroup`. The default location
can vary depending on the Linux distribution in use.|
+| CGroups mounted by YARN | If the system does not have CGroups mounted or it is mounted
to an inaccessible location then point `yarn.nodemanager.linux-container-executor.cgroups.mount-path`
to an empty directory. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to true.
A point to note here is that the container-executor binary will try to create and mount each
subsystem as a subdirectory under this path. If `cpu` is already mounted somewhere with `cpuacct`,
then the directory `cpu,cpuacct` will be created for the hierarchy.|
+| CGroups mounted already or linked but not in `/proc/mounts` | If cgroups is accessible
through lxcfs or simulated by another filesystem, then point `yarn.nodemanager.linux-container-executor.cgroups.mount-path`
to your CGroups root directory. Set `yarn.nodemanager.linux-container-executor.cgroups.mount`
to false. YARN tries to use this path first, before any CGroup mount point discovery. The
path should have a subdirectory for each CGroup hierarchy named by the comma separated CGroup
subsystems supported like `<path>/cpu,cpuacct`. Valid subsystem names are `cpu, cpuacct,
cpuset, memory, net_cls, blkio, freezer, devices`.|
+
 CGroups and security
 --------------------
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md
index 07c3765..f1308d5 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md
@@ -56,7 +56,7 @@ Following are the important interfaces:
 
 * Under very rare circumstances, programmer may want to directly use the 3 protocols to implement
an application. However, note that *such behaviors are no longer encouraged for general use
cases*.
 
-Writing a Simple Yarn Application
+Writing a Simple YARN Application
 ---------------------------------
 
 ### Writing a simple Client
@@ -574,4 +574,4 @@ Useful Links
 Sample Code
 -----------
 
-Yarn distributed shell: in `hadoop-yarn-applications-distributedshell` project after you
set up your development environment.
+YARN distributed shell: in `hadoop-yarn-applications-distributedshell` project after you
set up your development environment.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/47b145b9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md
index f5055d9..4973862 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md
@@ -84,7 +84,7 @@ container ID.
 
 ## The binding problem
 Hadoop YARN allows applications to run on the Hadoop cluster. Some of these are
-batch jobs or queries that can managed via Yarn’s existing API using its
+batch jobs or queries that can managed via YARN’s existing API using its
 application ID. In addition YARN can deploy ong-lived services instances such a
 pool of Apache Tomcat web servers or an Apache HBase cluster. YARN will deploy
 them across the cluster depending on the individual each component requirements
@@ -121,7 +121,7 @@ services accessible from within the Hadoop cluster
         /services/yarn
         /services/oozie
 
-Yarn-deployed services belonging to individual users.
+YARN-deployed services belonging to individual users.
 
         /users/joe/org-apache-hbase/demo1
         /users/joe/org-apache-hbase/demo1/components/regionserver1
@@ -148,7 +148,7 @@ their application master, to which they heartbeat regularly.
 
 ## Unsupported Registration use cases:
 
-1. A short-lived Yarn application is registered automatically in the registry,
+1. A short-lived YARN application is registered automatically in the registry,
 including all its containers. and unregistered when the job terminates.
 Short-lived applications with many containers will place excessive load on a
 registry. All YARN applications will be given the option of registering, but it
@@ -259,7 +259,7 @@ service since it supports many of the properties, We pick a part of the
ZK
 namespace to be the root of the service registry ( default: `yarnRegistry`).
 
 On top this base implementation we build our registry service API and the
-naming conventions that Yarn will use for its services. The registry will be
+naming conventions that YARN will use for its services. The registry will be
 accessed by the registry API, not directly via ZK - ZK is just an
 implementation choice (although unlikely to change in the future).
 
@@ -297,7 +297,7 @@ them.
 6. Core services will be registered using the following convention:
 `/services/{servicename}` e.g. `/services/hdfs`.
 
-7. Yarn services SHOULD be registered using the following convention:
+7. YARN services SHOULD be registered using the following convention:
 
         /users/{username}/{serviceclass}/{instancename}
 
@@ -823,8 +823,8 @@ The `RegistryPathStatus` class summarizes the contents of a node in the
registry
 ## Security
 
 The registry will allow a service instance can only be registered under the
-path where it has permissions. Yarn will create directories with appropriate
-permissions for users where Yarn deployed services can be registered by a user.
+path where it has permissions. YARN will create directories with appropriate
+permissions for users where YARN deployed services can be registered by a user.
 of the user account of the service instance. The admin will also create
 directories (such as `/services`) with appropriate permissions (where core Hadoop
 services can register themselves.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message