hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3119) Memory limit check need not be enforced unless aggregate usage of all containers is near limit
Date Tue, 16 Jun 2015 17:36:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588429#comment-14588429
] 

Chris Douglas commented on YARN-3119:
-------------------------------------

Systems that embrace more forgiving resource enforcement are difficult to tune, particularly
if those jobs run in multiple environments with different constraints (as is common when moving
from research/test to production). If jobs silently and implicitly use more resources than
requested, then users only learn that their container is under-provisioned when the cluster
workload shifts, and their pipelines start to fail.

I agree with [~aw]'s [feedback|https://issues.apache.org/jira/browse/YARN-3119?focusedCommentId=14303956&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14303956].
If this workaround is committed, this should be disabled by default and strongly discouraged.

> Memory limit check need not be enforced unless aggregate usage of all containers is near
limit
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3119
>                 URL: https://issues.apache.org/jira/browse/YARN-3119
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
>         Attachments: YARN-3119.prelim.patch
>
>
> Today we kill any container preemptively even if the total usage of containers for that
is well within the limit for YARN. Instead if we enforce memory limit only if the total limit
of all containers is close to some configurable ratio of overall memory assigned to containers,
we can allow for flexibility in container memory usage without adverse effects. This is similar
in principle to how cgroups uses soft_limit_in_bytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message