hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
Date Mon, 10 Apr 2017 12:19:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962766#comment-15962766
] 

Rajesh Balamohan commented on YARN-5764:
----------------------------------------

[~devaraj.k] Thank you for sharing the patch and the results.  Recent JVM versions have {{-XX:useNUMA}}
(java -XX:+PrintFlagsFinal | grep useNUMA). Enabling this would instruct JVM to be NUMA aware
and GC can take advantage of this fact. 

Was this flag ({{-XX:useNUMA}}) enabled in the tasks when running the benchmark? 

Hive on MR is outdated, network intensive and slow. It would be great, if BB benchmark can
be run with Hive on Tez which optimizes queries to a great extent. It has much better resource
utilization and also elimiates a lot of IO barriers and would be a lot efficient than MR codebase.

> NUMA awareness support for launching containers
> -----------------------------------------------
>
>                 Key: YARN-5764
>                 URL: https://issues.apache.org/jira/browse/YARN-5764
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, yarn
>            Reporter: Olasoji
>            Assignee: Devaraj K
>         Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance Results.pdf,
YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing costly remote
memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific
NUMA node and all subsequent memory allocations will be served by the same node, reducing
remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message