hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
Date Thu, 11 Dec 2014 21:27:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243171#comment-14243171

Craig Welch commented on YARN-2495:

Sorry if I'm jumping in late and asking redundant questions as a result, but I've gone through
the various related jiras and the design documents (incl updates) ( and the patch :-) ) and
I have some requirements related questions as a result.  Just a bit of background to make
sure I understand things - it appears that we've settled on two different but related features

1. To enable node labels to be added or removed for a given node without validation against
a centralized list of node labels (not on this patch, but relevant to the discussion) and
2. To enable node managers to specify their node labels based on local configuration and scripting
(this patch is specific to that feature).

These are, strictly speaking, orthogonal, but may be used together and will provide something
more in a 'combined feature'

A couple things about this feature (2) - I don't believe that it is necessary to add the node
label configuration to the local configuration (yarn-site) or the heartbeat as such to enable
configuration of labels for a node from the node in a decentralized fashion (e.g. a script
on the node saying "put these labels on me").  This can already be accomplished using the
admin cli from a script or calling a web service from the node (most likely the former, but
either is possible...), so I don't think we need this change to support the script case, it's
already possible to write a script to add a label to a node on the fly today without any changes.
 To make this dynamic we would need feature 1, from the above, but that's not covered in this
patch / is a separate discussion / and also does not require this change.  Also, I don't see
how this change allows a script to dynamically configure labels unless it was changing the
yarn-site or the like (I may have missed it, but I don't see that logic here) - and in any
case, it would not be necessary to add this logic to support that sort of configuration as
I pointed out.  Is this all just to support putting labels into the node-managers configuration
file and introducing them that way?  Do we have a solid need for that?  It's not needed for
the dynamic script case, which is all I've seen discussed here from a "requirements" perspective
(putting it into the config file / adding it to the heartbeat is implementation, I don't see
a requirement for it as such).

In a nutshell - do we need change 2 (this), or do we really just need change 1 (eliminating
validation of labels against a centralized list, at least as a configurable option)?

> Allow admin specify labels from each NM (Distributed configuration)
> -------------------------------------------------------------------
>                 Key: YARN-2495
>                 URL: https://issues.apache.org/jira/browse/YARN-2495
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, YARN-2495.20141030-1.patch,
YARN-2495.20141031-1.patch, YARN-2495.20141119-1.patch, YARN-2495.20141126-1.patch, YARN-2495.20141204-1.patch,
YARN-2495.20141208-1.patch, YARN-2495_20141022.1.patch
> Target of this JIRA is to allow admin specify labels in each NM, this covers
> - User can set labels in each NM (by setting yarn-site.xml (YARN-2923) or using script
suggested by [~aw] (YARN-2729) )
> - NM will send labels to RM via ResourceTracker API
> - RM will set labels in NodeLabelManager when NM register/update labels

This message was sent by Atlassian JIRA

View raw message