hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-951) Add hard minimum resource capabilities for container launching
Date Mon, 22 Jul 2013 23:24:49 GMT
Alejandro Abdelnur created YARN-951:
---------------------------------------

             Summary: Add hard minimum resource capabilities for container launching
                 Key: YARN-951
                 URL: https://issues.apache.org/jira/browse/YARN-951
             Project: Hadoop YARN
          Issue Type: Bug
          Components: nodemanager
    Affects Versions: 2.1.0-beta
            Reporter: Alejandro Abdelnur


This is a follow up of YARN-789, which enabled FairScheduler to handle zero capabilities resource
requests in one dimension (either zero CPU or zero memory).

When resource enforcement is enabled (cgroups for CPU and ProcfsBasedProcessTree for memory)
we cannot use zero because the underlying container processes will be killed.

We need to introduce an absolute or hard minimum:

* For CPU. Hard enforcement can be done via a cgroup cpu controller. Using an absolute minimum
of a few CPU shares (ie 10) in the LinuxContainerExecutor we ensure there is enough CPU cycles
to run the sleep process. This absolute minimum would only kick-in if zero is allowed, otherwise
will never kick in as the shares for 1 CPU are 1024.

* For Memory. Hard enforcement is currently done by the ProcfsBasedProcessTree.java, using
a minimum absolute of 1 or 2 MBs would take care of zero memory resources. And again, this
absolute minimum would only kick-in if zero is allowed, otherwise will never kick in as the
increment memory is in several MBs if not 1GB.

There would be no default for this hard minimum, if not set no correction will be done. If
set, then the MAX(hard-minimum, container-resource-capability) will be used. 

Effectively there will not be any impact unless the hard minimum capabilities are explicitly
set.

And, even if set, unless the scheduler is configured to allow zero capabilities, the hard-minimum
value will not kick in unless is set to a value higher than the MIN capabilities for a container.

Expected values, when set, would be 10 shares for CPU and 2 MB for memory.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message