hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
Date Tue, 10 Jan 2017 00:39:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813375#comment-15813375
] 

Wangda Tan commented on YARN-5764:
----------------------------------

Thanks [~devaraj.k] for updating design doc and patch, some questions/comments:

1) What is the benefit to manually specify NUMA node? Since this is potentially complex for
end user to specify, I think it's better to directly read data from OS.
2) Does the changes work on platform other than Linux?
3) I'm not quite sure about if this could happen: with this patch, YARN will launch process
one by one on each NUMA node to bind memory/cpu. Is it possible that there's another process
(outside of YARN) uses memory of NUMA node which causes processes launched by YARN failed
to bind or run?
4) This patch uses hard binding (get allocated resource on specified node or fail), is it
better to specify soft binding (prefer to allocate and can also accept other node). I think
soft binding should be default behavior to support NUMA.

Thoughts?

> NUMA awareness support for launching containers
> -----------------------------------------------
>
>                 Key: YARN-5764
>                 URL: https://issues.apache.org/jira/browse/YARN-5764
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, yarn
>            Reporter: Olasoji
>            Assignee: Devaraj K
>         Attachments: NUMA Awareness for YARN Containers.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing costly remote
memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific
NUMA node and all subsequent memory allocations will be served by the same node, reducing
remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message