hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4768) getAvailablePhysicalMemorySize can be inaccurate on linux
Date Tue, 08 Mar 2016 15:56:40 GMT

     [ https://issues.apache.org/jira/browse/YARN-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Nathan Roberts updated YARN-4768:
    Attachment: YARN-4768.patch

Patch for trunk.

Also changed getPhysicalMemorySize() to exclude:
- HardwareCorrupted pages - Not that uncommon.
- HugePagesTotal * hugePageSize - probably not commonly configured on compute nodes but just
in case it seems reasonable to not count these.

Comments welcome on alternative ways to approach these.

> getAvailablePhysicalMemorySize can be inaccurate on linux
> ---------------------------------------------------------
>                 Key: YARN-4768
>                 URL: https://issues.apache.org/jira/browse/YARN-4768
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0, 2.7.2
>         Environment: Linux
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-4768.patch
> Algorithm currently uses "MemFree" + "Inactive" from /proc/meminfo
> "Inactive" may not be a very good indication of how much memory can be readily freed
because it contains both:
> - Pages mapped with MAP_SHARED|MAP_ANONYMOUS (regardless of whether they're being actively
accessed or not. Unclear to me why this is the case...)
> - Pages mapped MAP_PRIVATE|MAP_ANONYMOUS that have not been accessed recently
> Both of these types of pages probably shouldn't be considered "Available".
> "Inactive(file)" would seem more accurate but it's not available in all kernel versions.
To keep things simple, maybe just use "Inactive(file)" if available, otherwise fallback to

This message was sent by Atlassian JIRA

View raw message