hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
Date Tue, 09 Jan 2018 00:02:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317379#comment-16317379
] 

Arun Suresh commented on YARN-6599:
-----------------------------------

[~leftnoteasy], did a slightly deeper dive.

This is my understanding of the general flow - I'd break this down the 2 phases.
# When the app makes the allocate call: This is where you do the updatePendingAsk etc for
each SchedulingRequest. This is also where you instantiate the AppPlacementAllocator and map
it to the request. Looks like the really interesting method is {{validateAndSetSchedulingRequest}},
which is where you validate the placement constraints. This method sets the valid targetAllocationTags
etc.
# When the node heartbeats: At this point, in the leafQueue, we pick the SchedulingRequest
with highest priority and finally, we call the {{canAllocate}} method which checks if the
Node can be assigned to the scheduling request based on the placementConstratint. right ?

Given the above, we should agree that:
This approach - while it ensures that allocation of a SchedulingRequest to a node will guarantee
that Constraints are NOT violated, It does nothing in the way of trying to FIND the nodes
that will meet the constraints.. agreed ?

My opinion - and something we should call out / document is that:
* If an app is more concerned about priority ordering of its requests, then we can recommend
using this approach.
* If the app the more concerned about an optimal placement, then the processor route will
give better results - since it activly tries to find nodes that satisfy constraints by considering
multiple schedulingRequests and multiple nodes.
Thuoghts ?

Also some other comments:
* In the Scheduler, you are adding a new List<SchedulingRequest> parameter to the allocate()
method. Do you think, a better approach might be to create a common superclass / interface
which both the SchedulingRequest and ResourceRequest extends and change the existing parameter
to {{List<? extends BaseRequest>}} ?
* If we do the above, then you wont have to duplicate methods like : application.updateResourceRequests
and normalizeSchedulingRequests


> Support rich placement constraints in scheduler
> -----------------------------------------------
>
>                 Key: YARN-6599
>                 URL: https://issues.apache.org/jira/browse/YARN-6599
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-6599-YARN-6592.003.patch, YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch,
YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, YARN-6599-YARN-6592.008.patch,
YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message