hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4719) Add a helper library to maintain node state and allows common queries
Date Tue, 01 Mar 2016 23:11:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174594#comment-15174594
] 

Karthik Kambatla commented on YARN-4719:
----------------------------------------

bq. For ClusterNodeTracker#nodes, can we use lock-free data structure to avoid copying the
whole set?
Not sure I understand the suggestion. Elaborate? 

bq. We'd better not add addBlacklistedNodeIdsToList to ClusterNodeTracker since it calls application's
logic, we should only include node related stuffs to ClusterNodeTracker.
I feel any logic that has to iterate through all nodes should go through ClusterNodeTracker
- that way, we don't run into cases where we access the list of nodes without a lock. Alternatively,
we could get a list of nodeIDs from ClusterNodeTracker and then look up individual nodes.
I am not particular about which approach, but also I don't quite see an issue with it being
a part of ClusterNodeTracker. Any particular reason you think this doesn't belong here? 

> Add a helper library to maintain node state and allows common queries
> ---------------------------------------------------------------------
>
>                 Key: YARN-4719
>                 URL: https://issues.apache.org/jira/browse/YARN-4719
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 2.8.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: yarn-4719-1.patch, yarn-4719-2.patch
>
>
> The scheduler could use a helper library to maintain node state and allowing matching/sorting
queries. Several reasons for this:
> # Today, a lot of the node state management is done separately in each scheduler. Having
a single library will take us that much closer to reducing duplication among schedulers.
> # Adding a filtering/matching API would simplify node labels and locality significantly.

> # An API that returns a sorted list for a custom comparator would help YARN-1011 where
we want to sort by allocation and utilization for continuous/asynchronous and opportunistic
scheduling respectively. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message