hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anubhav Dhoot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3119) Memory limit check need not be enforced unless aggregate usage of all containers is near limit
Date Tue, 03 Feb 2015 22:40:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304170#comment-14304170
] 

Anubhav Dhoot commented on YARN-3119:
-------------------------------------

Thanks for the comments.
Just wanted to give a quick response on the configurability. In my prelim patch I did add
the option to turn it off by default though i could use a better name (DEFAULT_TOTAL_MEM_USAGE_CHECK_ENABLED
is set to false). I also added a configurable threshold that lets you control how much of
the overall YARN container memory is allowed to be exceeded before we trigger the enforcement.
(Setting it to 0 would be another way to revert back to old behavior)

> Memory limit check need not be enforced unless aggregate usage of all containers is near
limit
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3119
>                 URL: https://issues.apache.org/jira/browse/YARN-3119
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
>         Attachments: YARN-3119.prelim.patch
>
>
> Today we kill any container preemptively even if the total usage of containers for that
is well within the limit for YARN. Instead if we enforce memory limit only if the total limit
of all containers is close to some configurable ratio of overall memory assigned to containers,
we can allow for flexibility in container memory usage without adverse effects. This is similar
in principle to how cgroups uses soft_limit_in_bytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message