hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
Date Tue, 26 Aug 2014 22:29:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111471#comment-14111471

Jason Lowe commented on YARN-2440:

For the case presented by [~sjlee0] the user has an 8 core system and wants to use at most
6 cores for YARN containers. That can be done by simply setting containers-cpu-percentage
to 75.  I don't see why we need a separate containers-cpu-cores parameter here, and I think
it causes more problems than it solves per my previous comment.  If we only want to support
whole-core granularity then I can see containers-cpu-cores as a better choice, but otherwise
containers-cpu-percentage is more flexible.

Also I don't see vcores being relevant for this JIRA.  The way vcores map to physical cores
is node-dependent, but apps ask for vcores in a node-independent fashion.  IIUC this JIRA
is focused on simply limiting the amount of CPU all YARN containers on the node can possibly
use in aggregate.  Changing the vcore-to-core ratio on the node will change how many containers
the node might run simultaneously, but it shouldn't impact how much of the physical CPU the
user wants reserved for non-container processes.

On a related note, it's interesting to step back and see if this is really what most users
will want in practice.  If the intent is to ensure the NM, DN, and other system processes
get enough CPU time then I think a better approach is to put those system processes in a peer
cgroup to the YARN containers cgroup and set their relative CPU shares accordingly.  Then
YARN containers can continue to use any spare CPU if desired (i.e.: no CPU "fragmentation")
but the system processes are guaranteed not to be starved out by the YARN containers.  Some
users may want a hard limit and hence why this feature would be useful for them, but I suspect
most users will not want to leave spare CPU lying around when containers need it.

bq. How about yarn.nodemanager.all-containers-cpu-cores and yarn.nodemanager.all-containers-cpu-percentage?

I'm indifferent on adding "all" as a prefix.  Something like yarn.nodemanager.containers-limit-cpu-percentage
might be more clear that this is a hard limit and CPUs can go idle even if containers are
demanding more from the machine than this limit.

> Cgroups should allow YARN containers to be limited to allocated cores
> ---------------------------------------------------------------------
>                 Key: YARN-2440
>                 URL: https://issues.apache.org/jira/browse/YARN-2440
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, apache-yarn-2440.2.patch,
> The current cgroups implementation does not limit YARN containers to the cores allocated
in yarn-site.xml.

This message was sent by Atlassian JIRA

View raw message