hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-951) Add hard minimum resource capabilities for container launching
Date Thu, 25 Jul 2013 23:49:50 GMT

    [ https://issues.apache.org/jira/browse/YARN-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720258#comment-13720258
] 

Bikas Saha commented on YARN-951:
---------------------------------

I would be happy to be totally wrong about this but I think we might be digging ourselves
deeper into a configuration hole. Just by reading the description I am confused about 0 resources
allowed but some minimum hard requirements are enforced as OS does not allow 0. Minimum will
come into play when 0 resources are allowed but not when 0 resources are not allowed. How
do understand this as a user? I am not suggesting that this hasnt been thought through or
there is no use for this etc. There are 2 paths 1) being really flexible and allowing all
possible definitions and combinations 2) restricting the parameters and allowing most of the
users to get stuff done easily. We seem to be going down the first path here.
                
> Add hard minimum resource capabilities for container launching
> --------------------------------------------------------------
>
>                 Key: YARN-951
>                 URL: https://issues.apache.org/jira/browse/YARN-951
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Alejandro Abdelnur
>            Assignee: Wei Yan
>
> This is a follow up of YARN-789, which enabled FairScheduler to handle zero capabilities
resource requests in one dimension (either zero CPU or zero memory).
> When resource enforcement is enabled (cgroups for CPU and ProcfsBasedProcessTree for
memory) we cannot use zero because the underlying container processes will be killed.
> We need to introduce an absolute or hard minimum:
> * For CPU. Hard enforcement can be done via a cgroup cpu controller. Using an absolute
minimum of a few CPU shares (ie 10) in the LinuxContainerExecutor we ensure there is enough
CPU cycles to run the sleep process. This absolute minimum would only kick-in if zero is allowed,
otherwise will never kick in as the shares for 1 CPU are 1024.
> * For Memory. Hard enforcement is currently done by the ProcfsBasedProcessTree.java,
using a minimum absolute of 1 or 2 MBs would take care of zero memory resources. And again,
this absolute minimum would only kick-in if zero is allowed, otherwise will never kick in
as the increment memory is in several MBs if not 1GB.
> There would be no default for this hard minimum, if not set no correction will be done.
If set, then the MAX(hard-minimum, container-resource-capability) will be used. 
> Effectively there will not be any impact unless the hard minimum capabilities are explicitly
set.
> And, even if set, unless the scheduler is configured to allow zero capabilities, the
hard-minimum value will not kick in unless is set to a value higher than the MIN capabilities
for a container.
> Expected values, when set, would be 10 shares for CPU and 2 MB for memory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message