hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
Date Mon, 25 Aug 2014 21:51:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109801#comment-14109801

Jason Lowe commented on YARN-2440:

Thanks for updating the patch, Varun.

I don't see why we need both a containers-cpu-cores and containers-cpu-percentage, and I think
it leads to confusion when both exist.  At first I did not realize that one overrode the other.
 Instead I assumed that if you set cpu-cores to X and cpu-percentage to Y then you were requesting
Y% of X cores.  Then there's the additional question of whether container usage is pinned
to those cores, etc.  Only having cpu-percentage is a simpler model that still allows the
user to specify cores indirectly (e.g.: 25% of an 8 core system is 2 cores).  Maybe I'm missing
the use case where we really need containers-cpu-cores and the confusing (to me at least)
override behavior between the two properties.

Other comments on the patch:

- I'm not thrilled about the name template "containers-cpu-*" since it could easily be misinterpreted
as a per-container thing as well, but I'm currently at a loss for a better prefix.  Suggestions
- Does getOverallLimits need to check for a quotaUS that's too low as well?
- I think minimally we need to log a warning if we're going to ignore setting up cgroups to
limit CPU usage across all containers if the user specified to do so.
- Related to the previous comment, I think it would be nice if we didn't try to setup any
limits if none were specified.  That way if there's some issue with correctly determining
the number of cores on a particular system it can still work in the default, "use everything"
- NodeManagerHardwareUtils.getContainerCores should be getContainersCores (the per-container
vs. all-containers confusion again)

> Cgroups should allow YARN containers to be limited to allocated cores
> ---------------------------------------------------------------------
>                 Key: YARN-2440
>                 URL: https://issues.apache.org/jira/browse/YARN-2440
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, apache-yarn-2440.2.patch,
> The current cgroups implementation does not limit YARN containers to the cores allocated
in yarn-site.xml.

This message was sent by Atlassian JIRA

View raw message