hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5849) Automatically create YARN control group for pre-mounted cgroups
Date Wed, 09 Nov 2016 19:22:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651809#comment-15651809
] 

Miklos Szegedi commented on YARN-5849:
--------------------------------------

I see now what you mean. The way the code behaves without mount before my patch is the following.
When we start initializing the resource chain, we look for the yarn cgroup in every(!) controller
and throw an exception and exit if the yarn group does not exist or not writable, regardless
of whether it is enabled or not.
{code}
  public enum CGroupController {
    CPU("cpu"),
    NET_CLS("net_cls"),
    BLKIO("blkio"),
    MEMORY("memory");
...  
  }
...
      for (CGroupController controller : CGroupController.values()) {
        String name = controller.getName();
        String controllerPath = findControllerInMtab(name, parsedMtab);

        if (controllerPath != null) {
          File f = new File(controllerPath + "/" + cGroupPrefix);

          if (FileUtil.canWrite(f)) {
            ret.put(controller, controllerPath);
          } else {
            String error =
                new StringBuffer("Mount point Based on mtab file: ")
                    .append(mtab)
                    .append(". Controller mount point not writable for: ")
                    .append(name).toString();

            LOG.error(error);
            throw new ResourceHandlerException(error);
          }
        } else {
          LOG.warn("Controller not mounted but automount disabled: " + name);
        }
      }
{code}
I will update my patch to avoid this and do the check and creation only, if the subsystem
in question is enabled.

> Automatically create YARN control group for pre-mounted cgroups
> ---------------------------------------------------------------
>
>                 Key: YARN-5849
>                 URL: https://issues.apache.org/jira/browse/YARN-5849
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>            Reporter: Miklos Szegedi
>            Assignee: Miklos Szegedi
>            Priority: Minor
>         Attachments: YARN-5849.000.patch, YARN-5849.001.patch
>
>
> Yarn can be launched with linux-container-executor.cgroups.mount set to false. It will
search for the cgroup mount paths set up by the administrator parsing the /etc/mtab file.
You can also specify resource.percentage-physical-cpu-limit to limit the CPU resources assigned
to containers.
> linux-container-executor.cgroups.hierarchy is the root of the settings of all YARN containers.
If this is specified but not created YARN will fail at startup:
> Caused by: java.io.FileNotFoundException: /cgroups/cpu/hadoop-yarn/cpu.cfs_period_us
(Permission denied)
> org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler.updateCgroup(CgroupsLCEResourcesHandler.java:263)
> This JIRA is about automatically creating YARN control group in the case above. It reduces
the cost of administration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message