hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
Date Wed, 10 Sep 2014 18:13:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128843#comment-14128843

Vinod Kumar Vavilapalli commented on YARN-2440:

bq. As I mentioned before, I think most users would rather not use the functionality proposed
by this JIRA but instead setup peer cgroups for other systems and set their relative cgroup
shares appropriately. With this JIRA the CPUs could sit idle despite demand from YARN containers,
while a peer cgroup setup allows CPU guarantees without idle CPUs if the demand is there.
[~jlowe], agree with the general philosophy. Though we are not yet there in practice - datanodes,
region servers don't yet live in cgroups in many sites. Looking back at this JIRA, I see a
good use for this. Having the overall YARN limit will help ensure that apps' containers don't
thrash cpu once we start enabling support.

The other dimension to this is determinism w.r.t performance. Limiting to allocated cores
overall (as well as per container later) helps orgs run workloads and reason about them deterministically.
One of the examples is benchmarking apps, but deterministic execution is a desired option
beyond benchmarks too.

> Cgroups should allow YARN containers to be limited to allocated cores
> ---------------------------------------------------------------------
>                 Key: YARN-2440
>                 URL: https://issues.apache.org/jira/browse/YARN-2440
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, apache-yarn-2440.2.patch,
apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, apache-yarn-2440.5.patch, apache-yarn-2440.6.patch,
> The current cgroups implementation does not limit YARN containers to the cores allocated
in yarn-site.xml.

This message was sent by Atlassian JIRA

View raw message