hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4719) Add a helper library to maintain node state and allows common queries
Date Mon, 07 Mar 2016 05:10:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182591#comment-15182591

Wangda Tan commented on YARN-4719:

Hi [~kasha],
bq. through addNode and removeNode so total_cluster_resources, total_inflated_cluster_resources
(for YARN-1011), max_cluster_resources are not affected by other scheduler code.
I may not understand about this, could you elaborate?

For handle scheduler code to iterate nodes, we could either:
# Use concurrent map to avoid locking, and code will not break. Drawback: we need to handle
stale data.
# Expose lock to external caller, so scheduler can get readlock of ClusterNodeTracker and
do iteration. Drawback: iteration nodes and allocating containers could lock ClusterNodeTracker
for long time.
# Assume synchronize lock of scheduler will be acquired when make changes to ClusterNodeTracker
(like addNode, removeNode, etc.), and also when iterating nodes. We don't need extra lock
of returned node collections. Drawback: this hides locks to external caller behaviors, and
in the future scheduler could remove synchronized lock to get better performance.

I would suggest to look at if #1 is doable (handle stale data and assumes eventually consistency).
#1 should have best performance and flexible to future scheduler changes.

> Add a helper library to maintain node state and allows common queries
> ---------------------------------------------------------------------
>                 Key: YARN-4719
>                 URL: https://issues.apache.org/jira/browse/YARN-4719
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 2.8.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: yarn-4719-1.patch, yarn-4719-2.patch, yarn-4719-3.patch
> The scheduler could use a helper library to maintain node state and allowing matching/sorting
queries. Several reasons for this:
> # Today, a lot of the node state management is done separately in each scheduler. Having
a single library will take us that much closer to reducing duplication among schedulers.
> # Adding a filtering/matching API would simplify node labels and locality significantly.

> # An API that returns a sorted list for a custom comparator would help YARN-1011 where
we want to sort by allocation and utilization for continuous/asynchronous and opportunistic
scheduling respectively. 

This message was sent by Atlassian JIRA

View raw message