hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
Date Tue, 15 Aug 2017 23:26:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128055#comment-16128055

Wangda Tan commented on YARN-5764:


Thanks for updating the patch, I checked the latest patch implementation. Some suggestions:

1) It added numa controller for both default container executor and linux container executor,
does it make sense to use this feature under default container executor since CPU asks might
be ignored in RM side (so asking 100 vcores is same as asking 1 vcores).

2) If we don't have to add support of DefaultContainerExecutor, probably we can leverage the
latest ResourceHandlerModule, with that we can easier plug the numa related logics.

3) It seems this patch doesn't handle NM restart recovery. I think we need to recover what
allocated by NM.

Probably you can take a look at approach of https://issues.apache.org/jira/browse/YARN-6620,
and some common libraries added in YARN-6620 (such as NM resource recovery) could be used
to implement this feature.

+ [~shanekumpf@gmail.com].

> NUMA awareness support for launching containers
> -----------------------------------------------
>                 Key: YARN-5764
>                 URL: https://issues.apache.org/jira/browse/YARN-5764
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, yarn
>            Reporter: Olasoji
>            Assignee: Devaraj K
>         Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance Results.pdf,
YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch, YARN-5764-v3.patch
> The purpose of this feature is to improve Hadoop performance by minimizing costly remote
memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific
NUMA node and all subsequent memory allocations will be served by the same node, reducing
remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message