hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
Date Sun, 20 Jul 2014 20:43:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068043#comment-14068043
] 

Allen Wittenauer commented on YARN-796:
---------------------------------------

I agree pretty much completely with everything Sandy said, especially on the centralized configuration.
 It actually makes configuration harder for heterogeneous node setups.

One caveat:

{code}
I'm also skeptical about the need for adding/removing labels dynamically. Do we have concrete
use cases for this?
{code}

If you have the nodemanager push the labels to the RM (esp if you can do this via user defined
script or java class...), you basically have to have dynamic labels for nodes.  Use cases
are pretty easy to hit if you label nodes based upon the software stack installed. A quick
example for those not following:

# User writes software that depends upon a particular version of libfoo.so.2.
# Configuration management does an install of  libfoo.so.2
# NodeManager label script picks up that it has both libfoo.so.1 and libfoo.so.2.  Publishes
that it now has "libfoo1" and "libfoo2".  (Remember, this is C and not the screwed up Java
universe so having two versions is completely legitimate)
# system can now do operations appropriate for either libfoo on that node.
# libfoo1 gets deprecated and removed from the system, again via configuration management.
# label script picks up change and removes libfoo1 from label listing
# system acts appropriately and no longer does operations on node based upon libfoo1 label

... and all without restarting or reconfiguring anything on the Hadoop side.  If there is
any sort of manual step required in configuration the nodes short of the initial label script/class
and other obviously user-provided bits, then we've failed.

> Allow for (admin) labels on nodes and resource-requests
> -------------------------------------------------------
>
>                 Key: YARN-796
>                 URL: https://issues.apache.org/jira/browse/YARN-796
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Wangda Tan
>         Attachments: LabelBasedScheduling.pdf, Node-labels-Requirements-Design-doc-V1.pdf,
YARN-796.patch
>
>
> It will be useful for admins to specify labels for nodes. Examples of labels are OS,
processor architecture etc.
> We should expose these labels and allow applications to specify labels on resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message