hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
Date Wed, 05 Apr 2017 05:29:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956325#comment-15956325

Wangda Tan commented on YARN-6443:


I was thinking about similar things while working on YARN-5139. One issue I can think of is,
it will break ordering of applications/queues/schedulerKeys.

The ideal solution in my mind is pass multiple nodes while doing scheduling and scoring/sorting
nodes based on placement policy (for example, based on locality).

There're some optimizations we can make to reduce the overhead of sorting. I put some of the
stuffs in early patch of YARN-5139: https://issues.apache.org/jira/secure/attachment/12825346/wip-4.YARN-5139.patch#file-36
please check LocalityNodesScorer.java. 

I think we can use the similar approach to optimize affinity/anti-affinity which I mentioned

Please let me know your thoughts.

> Allow for Priority order relaxing in favor of improved node/rack locality 
> --------------------------------------------------------------------------
>                 Key: YARN-6443
>                 URL: https://issues.apache.org/jira/browse/YARN-6443
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacity scheduler, fairscheduler
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
> Currently the Schedulers examine an applications pending Requests in Priority order.
This JIRA proposes to introduce a flag (either via the ApplicationMasterService::registerApplication()
or via some Scheduler configuration) to favor an ordering that is baised to the node that
is currently heartbeating by relaxing the priority constraint.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message