hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-398) Allow white-list and black-list of resources
Date Tue, 26 Feb 2013 01:20:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586576#comment-13586576

Arun C Murthy commented on YARN-398:

Here is an approach I have a prelim patch on which I'll put up shortly...

Essentially, we need an extra (boolean?) field per ResourceRequest (RR) called 'blacklist'
or, essentially, 'do not fall through'.

What this provides is a way to tell the Scheduler to stop it's current fall-through scheduling
mechanism of node-to-rack-ANY ...

When the flag is set, the scheduler should *not* fall-through and look for rack-locality or

Example scenarios: 
# if you want to blacklist a node, set the flag on RR of the node
# if you want to blacklist a rack, set the flag on the RR of the rack (obviously do not send
any RRs for any nodes in the rack)
# if you want to whitelist a node (or a set of them): blacklist ANY and either don't pass
other rack RRs or blacklist all rack RRs (for those nodes you care about).

Obviously, this works across priorities since each RR is per-priority.

Essentially, this flag provides the ability for the application to ask the scheduler to disable
it's 'locality relaxation' or 'fall-through'.


This way, we don't radically re-design our protocol in YARN v1 (i.e. hadoop-2.x) and yet provide
blacklist/whitelist for individual and a set of nodes/racks.

Clearly, we can make this easier for end-users by adding a simple client api which takes care
by providing apis like: blacklistNodes/whitelistNodes and does it automatically for each priority
simplifying the application writer's life.


Thoughts? Do folks see any long-term (i.e. hadoop-2.x), negative repercussions due to this?
> Allow white-list and black-list of resources
> --------------------------------------------
>                 Key: YARN-398
>                 URL: https://issues.apache.org/jira/browse/YARN-398
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
> Allow white-list and black-list of resources in scheduler api.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message