hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
Date Sat, 10 Mar 2018 01:06:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393910#comment-16393910

Wangda Tan commented on YARN-5764:


Just took a very quick look at overall integration to other NM components, some comments:
1) The latest patch doesn't persistent assigned NUMA resources, you can take a look at GpuResourceAllocator
as an example: 
          // Update state store.
          nmContext.getNMStateStore().storeAssignedResources(container, GPU_URI,
              new ArrayList<>(assignedGpus));

2) Is it better to move all NUMA related works to {{org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.numa}}?
Which is consistent to other plugins such as GPU/FPGA. The scheduler package is for new container
allocation. (Just like scheduler module in RM). 

> NUMA awareness support for launching containers
> -----------------------------------------------
>                 Key: YARN-5764
>                 URL: https://issues.apache.org/jira/browse/YARN-5764
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, yarn
>            Reporter: Olasoji
>            Assignee: Devaraj K
>            Priority: Major
>         Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance Results.pdf,
YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch, YARN-5764-v3.patch, YARN-5764-v4.patch,
YARN-5764-v5.patch, YARN-5764-v6.patch, YARN-5764-v7.patch, YARN-5764-v8.patch, YARN-5764-v9.patch
> The purpose of this feature is to improve Hadoop performance by minimizing costly remote
memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific
NUMA node and all subsequent memory allocations will be served by the same node, reducing
remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message