hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ferguson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
Date Sat, 16 Jun 2012 00:23:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13296036#comment-13296036

Andrew Ferguson commented on MAPREDUCE-4334:

Hi Arun,

I've thought some more about implementing taskset since our chat at the YARN meet-up.

One benefit of cgroups is they're "set it and forget it" -- in the ContainerExecutor, we simply
place the new task in the appropriate cgroup, and the kernel will take care of the rest. This
would allow us to ditch the ContainersMonitor infrastructure.

On the other hand, with taskset, we will need to do the CPU scheduling ourselves. Say I have
two cores and start with two processes, A (requested 0.5 cores) and B (requested 0.5 cores).
I can start by putting them both on core 1 for efficiency, or I can put them on separate cores
for higher utilization. But if process C (requested 1 core) comes along, I will need to set
A & B to the same core. This is just a simple scenario, but more cores and processes will
likely grow a complicated CPU scheduler inside the NodeManager (ContainersMonitorImpl is probably
the right place, since it is already monitoring container resource usage).

tl;dr -- I believe cgroups requires only local state when launching containers, while taskset
requires us to maintain global state.


> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>                 Key: MAPREDUCE-4334
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
> Once we get in MAPREDUCE-4327, it will be important to actually enforce limits on CPU
consumption of containers. 
> Several options spring to mind:
> # taskset (RHEL5+)
> # cgroups (RHEL6+)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message