hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ferguson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
Date Wed, 20 Jun 2012 19:22:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397794#comment-13397794

Andrew Ferguson commented on MAPREDUCE-4351:

Hi Arun,

I initially considered that approach (having ContainersMonitor go away), but decided against
it for the following reasons:

- Because the initial configuration of cgroups requires root access, some users may not be
able to use them. Therefore, it makes sense to keep the current ContainersMonitor, which monitors
memory usage and does its own OOM killing.

- Maintaining the separation between the ContainerExecutor (which just does launching) and
the ContainersMonitor (which just does enforcement) lowers the amount of code overlap for
supporting each permutation: secure executor with/without cgroups, default executor with/without
cgroups, a ContainersMonitor with taskset for CPU instead of cgroups, etc.

- With this approach, the ContainersMonitor continues to receive the ContainerStopMonitoringEvent
event, which allows it to delete cgroups which are no longer in use.

- Finally, the current ContainersMonitor does not start enforcing the memory limit until after
a short delay because of the potential to double-count resources when the JVM does a fork()+exec()
-- see comment above isProcessTreeOverLimit().  Keeping resource enforcement separate from
process launching gives us the flexibility to delay the start of enforcement if needed.

do these reasons make sense?

I'm currently writing a "CgroupsContainersMonitor" to complement the "DefaultContainersMonitor,"
and which plugs-in using this patch.  As you point out, this CgroupsContainerMonitor doesn't
do much work: it just configures cgroups and places processes in them, allowing the kernel
to do the actual enforcement work.

> Make ContainersMonitor pluggable
> --------------------------------
>                 Key: MAPREDUCE-4351
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2, nodemanager
>            Reporter: Andrew Ferguson
>            Assignee: Andrew Ferguson
>         Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, MAPREDUCE-4351-v3.patch,
MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch
> Make the existing ContainersManager pluggable, just as the ContainerExecutor is currently.
This will allow us to add container resource enforcement using other techniques (such as cgroups)
in an extensible fashion.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message