hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-951) Add hard minimum resource capabilities for container launching
Date Fri, 26 Jul 2013 01:41:49 GMT

    [ https://issues.apache.org/jira/browse/YARN-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720344#comment-13720344

Arun C Murthy commented on YARN-951:

I'm as confused as Bikas.

I thought with YARN-789 FS can now set minimum to 1 (or 10, or 120) and multiplier to 1024
and we are done. Why do we need anything else?
> Add hard minimum resource capabilities for container launching
> --------------------------------------------------------------
>                 Key: YARN-951
>                 URL: https://issues.apache.org/jira/browse/YARN-951
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Alejandro Abdelnur
>            Assignee: Wei Yan
> This is a follow up of YARN-789, which enabled FairScheduler to handle zero capabilities
resource requests in one dimension (either zero CPU or zero memory).
> When resource enforcement is enabled (cgroups for CPU and ProcfsBasedProcessTree for
memory) we cannot use zero because the underlying container processes will be killed.
> We need to introduce an absolute or hard minimum:
> * For CPU. Hard enforcement can be done via a cgroup cpu controller. Using an absolute
minimum of a few CPU shares (ie 10) in the LinuxContainerExecutor we ensure there is enough
CPU cycles to run the sleep process. This absolute minimum would only kick-in if zero is allowed,
otherwise will never kick in as the shares for 1 CPU are 1024.
> * For Memory. Hard enforcement is currently done by the ProcfsBasedProcessTree.java,
using a minimum absolute of 1 or 2 MBs would take care of zero memory resources. And again,
this absolute minimum would only kick-in if zero is allowed, otherwise will never kick in
as the increment memory is in several MBs if not 1GB.
> There would be no default for this hard minimum, if not set no correction will be done.
If set, then the MAX(hard-minimum, container-resource-capability) will be used. 
> Effectively there will not be any impact unless the hard minimum capabilities are explicitly
> And, even if set, unless the scheduler is configured to allow zero capabilities, the
hard-minimum value will not kick in unless is set to a value higher than the MIN capabilities
for a container.
> Expected values, when set, would be 10 shares for CPU and 2 MB for memory.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message