hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sidharta Seethana (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1856) cgroups based memory monitoring for containers
Date Sat, 05 Dec 2015 00:54:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042505#comment-15042505
] 

Sidharta Seethana commented on YARN-1856:
-----------------------------------------

Hi [~vvasudev], 

Thanks for the patch. Please see below for some comments.

{code}
@VisibleForTesting
static final String HARD_LIMIT = "limit_in_bytes";
@VisibleForTesting
static final String SOFT_LIMIT = "soft_limit_in_bytes";
@VisibleForTesting
static final String SWAPPINESS = "swappiness”;
{code}

Could you move these constants to CGroupsHandler ? There are already some cgroups parameter
constants defined there.

{code}
if (configuration.getBoolean(YarnConfiguration.NM_PMEM_CHECK_ENABLED,
    YarnConfiguration.DEFAULT_NM_PMEM_CHECK_ENABLED)) {
  LOG.warn("You have enabled the default YARN physical memory health"
      + " checker as well as the CGroups memory controller. This could"
      + " lead to unpredictable behaviour");
}
{code}

IMO, If the behavior here is unpredictable, we should simply error our here in case both are
enabled. 

{code}
long softLimit = (long) (container.getResource().getMemory() * 0.90f);
{code}

We should make the fraction configurable, I think. What are the implications of the soft limit?

{code}
public static final String NM_MEMORY_RESOURCE_PREFIX = NM_PREFIX
    + "resource.memory.";

public static final String NM_MEMORY_RESOURCE_ENABLED =
    NM_MEMORY_RESOURCE_PREFIX + "enabled";
public static final boolean DEFAULT_NM_MEMORY_RESOURCE_ENABLED = false;

public static final String NM_MEMORY_RESOURCE_CGROUPS_SWAPPINESS =
    NM_MEMORY_RESOURCE_PREFIX + "cgroups.swappiness";
public static final int DEFAULT_NM_MEMORY_RESOURCE_CGROUPS_SWAPPINESS = 0;
{code}

Since we are skipping changes to yarn-default.xml (based on changes I see in TestYarnConfigurationFields),
these should be marked @Private , similar to how network/disk configs settings are annotated?


Thinking aloud here : should we add support in some form for {{memory.oom_control}} and notifications/stats?

> cgroups based memory monitoring for containers
> ----------------------------------------------
>
>                 Key: YARN-1856
>                 URL: https://issues.apache.org/jira/browse/YARN-1856
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 2.3.0
>            Reporter: Karthik Kambatla
>            Assignee: Varun Vasudev
>         Attachments: YARN-1856.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message