hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ferguson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3) Add support for CPU isolation/monitoring of containers
Date Thu, 09 Aug 2012 05:44:18 GMT

     [ https://issues.apache.org/jira/browse/YARN-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Andrew Ferguson updated YARN-3:

    Attachment: YARN-3-lce_only-v1.patch

This patch augments the LinuxContainerExecutor with an LCEResourcesHandler, which can be used
to enforce resource limits using either cgroups (in this patch) or sched_setaffinity/taskset
(future patch). A DefaultLCEResourcesHandler is also provided which does not enforce any new
resource limits.

The LCEResourcesHandler interface (and concrete classes) are introduced to keep the LinuxContainerExecutor
java class simple, and to separate the cgroups and future sched_setaffinity logic.

The resources handler code is split across the introduced Java classes, and the existing container-executor
native binary. This is done to minimize the amount of code added to the native binary, to
provide easy logging via Java mechanisms, and because a singleton is needed to track CPU assignments
when using sched_setaffinity/taskset.

The handler operates synchronously with the execution of the container.

The changes to the native code are:
1) A resources option has been added to the LaunchContainer command. This option is used to
convey a list of cgroups into which the container should be placed before the user command
is launched, and, in the future, will be used to alternatively covey a list of CPUs which
the process should be pinned to, if using sched_setaffinity instead of cgroups.

2) A --mount-cgroups command has been added to the native code. This command will mount cgroups
controllers and create hierarchies for the NodeManager to manage. This feature is optional
(see below), and exposed to the Java code via a new method in LinuxContainerExecutor.java.

The following configuration options are introduced:

yarn.nodemanager.linux-container-executor.resources-handler.class -- The class which should
assist the LCE in handling resources.

yarn.nodemanager.linux-container-executor.cgroups.hierarchy -- The cgroups hierarchy under
which to place YARN proccesses (cannot contain commas). Only used when the LCE resources handler
is set to the CgroupsLCEResourcesHandler.

yarn.nodemanager.linux-container-executor.cgroups.mount -- Whether the LCE should attempt
to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.

yarn.nodemanager.linux-container-executor.cgroups.mount-path -- Where the LCE should attempt
to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup. The path
must exist before the NodeManager is launched. Only used when the LCE resources handler is
set to the CgroupsLCEResourcesHandler.
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>                 Key: YARN-3
>                 URL: https://issues.apache.org/jira/browse/YARN-3
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Andrew Ferguson
>         Attachments: MAPREDUCE-4334-executor-v1.patch, MAPREDUCE-4334-executor-v2.patch,
MAPREDUCE-4334-executor-v3.patch, MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch,
MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, MAPREDUCE-4334-pre3-with_cpu.patch,
MAPREDUCE-4334-pre3.patch, MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, YARN-3-lce_only-v1.patch,
mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message