hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ferguson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
Date Mon, 18 Jun 2012 21:18:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396261#comment-13396261
] 

Andrew Ferguson commented on MAPREDUCE-4334:
--------------------------------------------

ok, putting all of this in the ContainerExecutor is not the way to go, as it precludes use
of secure Hadoop's Linux container-executor.

In my new design, ContainerMonitor will be a pluggable component, just as ContainerExecutor
is now. Then, we can provide a ContainerMonitor which uses cgroups to control resource usage,
rather than the existing ContainerMonitor (to be renamed as "DefaultContainerMonitor"). This
has several advantages:
1) allows us to keep existing ContainerMonitor for users who can't use cgroups (eg, users
without root access during Hadoop setup)
2) ContainerMonitor already receives an event when it's time to stop monitoring, which we
can use as notification to delete the container's cgroup
3) ContainerMonitor receives the resource limits already; no need to calculate them based
on the configs
4) A pluggable ContainerMonitor paves the way for ContainerMonitors on other platforms

I will first open a sub-task to make ContainerMonitor pluggable.

The only trouble spot with this design is that it's not possible to move another non-root
user's process into a cgroup. I plan to extend the secure container-executor to be able to
make such a move.

Please let me know if you have any feedback about this proposal.


thank you,
Andrew
                
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-4334
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> Once we get in MAPREDUCE-4327, it will be important to actually enforce limits on CPU
consumption of containers. 
> Several options spring to mind:
> # taskset (RHEL5+)
> # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message