Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 9 Nov 2016 19:22:59 +0000 (UTC)
From: "Miklos Szegedi (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13019005.1478554169000.235623.1478719379581@Atlassian.JIRA>
In-Reply-To: <JIRA.13019005.1478554169000@Atlassian.JIRA>
References: <JIRA.13019005.1478554169000@Atlassian.JIRA> <JIRA.13019005.1478554169045@arcas>
Subject: [jira] [Commented] (YARN-5849) Automatically create YARN control
 group for pre-mounted cgroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 09 Nov 2016 19:23:01 -0000


    [ https://issues.apache.org/jira/browse/YARN-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651809#comment-15651809 ] 

Miklos Szegedi commented on YARN-5849:
--------------------------------------

I see now what you mean. The way the code behaves without mount before my patch is the following. When we start initializing the resource chain, we look for the yarn cgroup in every(!) controller and throw an exception and exit if the yarn group does not exist or not writable, regardless of whether it is enabled or not.
{code}
  public enum CGroupController {
    CPU("cpu"),
    NET_CLS("net_cls"),
    BLKIO("blkio"),
    MEMORY("memory");
...  
  }
...
      for (CGroupController controller : CGroupController.values()) {
        String name = controller.getName();
        String controllerPath = findControllerInMtab(name, parsedMtab);

        if (controllerPath != null) {
          File f = new File(controllerPath + "/" + cGroupPrefix);

          if (FileUtil.canWrite(f)) {
            ret.put(controller, controllerPath);
          } else {
            String error =
                new StringBuffer("Mount point Based on mtab file: ")
                    .append(mtab)
                    .append(". Controller mount point not writable for: ")
                    .append(name).toString();

            LOG.error(error);
            throw new ResourceHandlerException(error);
          }
        } else {
          LOG.warn("Controller not mounted but automount disabled: " + name);
        }
      }
{code}
I will update my patch to avoid this and do the check and creation only, if the subsystem in question is enabled.

> Automatically create YARN control group for pre-mounted cgroups
> ---------------------------------------------------------------
>
>                 Key: YARN-5849
>                 URL: https://issues.apache.org/jira/browse/YARN-5849
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>            Reporter: Miklos Szegedi
>            Assignee: Miklos Szegedi
>            Priority: Minor
>         Attachments: YARN-5849.000.patch, YARN-5849.001.patch
>
>
> Yarn can be launched with linux-container-executor.cgroups.mount set to false. It will search for the cgroup mount paths set up by the administrator parsing the /etc/mtab file. You can also specify resource.percentage-physical-cpu-limit to limit the CPU resources assigned to containers.
> linux-container-executor.cgroups.hierarchy is the root of the settings of all YARN containers. If this is specified but not created YARN will fail at startup:
> Caused by: java.io.FileNotFoundException: /cgroups/cpu/hadoop-yarn/cpu.cfs_period_us (Permission denied)
> org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler.updateCgroup(CgroupsLCEResourcesHandler.java:263)
> This JIRA is about automatically creating YARN control group in the case above. It reduces the cost of administration.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org