hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MENG DING (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4599) Set OOM control for memory cgroups
Date Tue, 02 Feb 2016 23:11:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129300#comment-15129300

MENG DING commented on YARN-4599:

I have a question regarding when OOM controller will kick in. Is it true that:

* If memory.swappiness is set to 0, then OOM controller will kick in when  {{memory.limit_in_bytes}}
(hard limit) is reached.
* If memory.swappiness is not set to 0, then OOM controller will kick in only when all available
swap in the system is used up, as {{memory.memsw.limit_in_bytes}} is not a configurable parameter
in YARN right now?

Looking at this link, I am a little bit confused: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

bq. Consider the following example: setting memory.limit_in_bytes = 2G and memory.memsw.limit_in_bytes
= 4G for a certain cgroup will allow processes in that cgroup to allocate 2 GB of memory and,
once exhausted, allocate another 2 GB of swap only. The memory.memsw.limit_in_bytes parameter
represents the sum of memory and swap. Processes in a cgroup that does not have the memory.memsw.limit_in_bytes
parameter set can potentially use up all the available swap (after exhausting the set memory
limitation) and trigger an Out Of Memory situation caused by the lack of available swap.

bq. *memory.swappiness* Note that a value of 0 does not prevent process memory being swapped
out; swap out might still happen when there is a shortage of system memory because the global
virtual memory management logic does not read the cgroup value.

> Set OOM control for memory cgroups
> ----------------------------------
>                 Key: YARN-4599
>                 URL: https://issues.apache.org/jira/browse/YARN-4599
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.9.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
> YARN-1856 adds memory cgroups enforcing support. We should also explicitly set OOM control
so that containers are not killed as soon as they go over their usage. Today, one could set
the swappiness to control this, but clusters with swap turned off exist.

This message was sent by Atlassian JIRA

View raw message