Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 15 Aug 2017 23:26:00 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13014081.1477003925000.38245.1502839560799@Atlassian.JIRA>
In-Reply-To: <JIRA.13014081.1477003925000@Atlassian.JIRA>
References: <JIRA.13014081.1477003925000@Atlassian.JIRA> <JIRA.13014081.1477003925806@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-5764) NUMA awareness support for launching
 containers
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 15 Aug 2017 23:26:09 -0000


    [ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128055#comment-16128055 ] 

Wangda Tan commented on YARN-5764:
----------------------------------

[~devaraj.k], 

Thanks for updating the patch, I checked the latest patch implementation. Some suggestions:

1) It added numa controller for both default container executor and linux container executor, does it make sense to use this feature under default container executor since CPU asks might be ignored in RM side (so asking 100 vcores is same as asking 1 vcores).

2) If we don't have to add support of DefaultContainerExecutor, probably we can leverage the latest ResourceHandlerModule, with that we can easier plug the numa related logics.

3) It seems this patch doesn't handle NM restart recovery. I think we need to recover what allocated by NM.

Probably you can take a look at approach of https://issues.apache.org/jira/browse/YARN-6620, and some common libraries added in YARN-6620 (such as NM resource recovery) could be used to implement this feature.

+ [~shanekumpf@gmail.com].

> NUMA awareness support for launching containers
> -----------------------------------------------
>
>                 Key: YARN-5764
>                 URL: https://issues.apache.org/jira/browse/YARN-5764
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, yarn
>            Reporter: Olasoji
>            Assignee: Devaraj K
>         Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch, YARN-5764-v3.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing costly remote memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific NUMA node and all subsequent memory allocations will be served by the same node, reducing remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org