hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dian Fu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2923) Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup
Date Mon, 29 Jun 2015 07:40:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605238#comment-14605238

Dian Fu commented on YARN-2923:

Hi [~Naganarasimha],
Sorry to come to this JIRA late. Thanks a lot for working on this. Most codes LGTM. Just have
the following rough thoughts:
1. Classes such as {{AbstractNodeLabelsProvider}} are located in directory {{hadoop-yarn-server-nodemanager}},
I'm think if we can move it to {{hadoop-yarn-server-common}}. This would ease the work of
adding similar node label provider mechanism at RM side in the future if possible. There may
need some minor code reactor work to do, such as it accesses NM side configuration {{NM_NODE_LABELS_PROVIDER_FETCH_INTERVAL_MS}}.
2. For YARN-2923 and YARN-2729, both {{ScriptBasedNodeLabelsProvider}} and {{ConfigurationNodeLabelsProvider}}
have a mechanism to periodically execute a script or read configuration to fetch labels. Is
it possible to merge {{ScriptBasedNodeLabelsProvider}} and {{ConfigurationNodeLabelsProvider}}
into one class? This class can provide the function to periodically fetch labels and the actual
work of how to fetch labels can be configured in a plugable way. The benefits of this approach
are: a) remove duplicate code in {{ScriptBasedNodeLabelsProvider}} and {{ConfigurationNodeLabelsProvider}};
b) ease third parts to add their own node label provider.

Not sure if the above thoughts are clear and make sense to you? I can help to do some POC
based on YARN-2923 and YARN-2729 if needed.

> Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration
> -----------------------------------------------------------------------------------------------------
>                 Key: YARN-2923
>                 URL: https://issues.apache.org/jira/browse/YARN-2923
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>             Fix For: 2.8.0
>         Attachments: YARN-2923.20141204-1.patch, YARN-2923.20141210-1.patch, YARN-2923.20150328-1.patch,
YARN-2923.20150404-1.patch, YARN-2923.20150517-1.patch
> As part of Distributed Node Labels configuration we need to support Node labels to be
configured in Yarn-site.xml. And on modification of Node Labels configuration in yarn-site.xml,
NM should be able to get modified Node labels from this NodeLabelsprovider service without
NM restart

This message was sent by Atlassian JIRA

View raw message