hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7612) Add Placement Processor Framework
Date Thu, 21 Dec 2017 16:18:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300223#comment-16300223

Arun Suresh commented on YARN-7612:

Thanks for taking a look [~cheersyang]

bq. If a request has rack max cardinality 3, that means the algorithm output will only have
3 candidate nodes, what if it 1 of them gets rejected in the scheduler phase? 
This is why we have a retry for the scheduler phase. A request rejected by the Scheduler is
funneled back to the PlacementAlgortihm with an incremented placementAttempt. The algorithm
can then decide to place it on another node (That reminds me - the algorithm input can also
have a list of blacklisted nodes, nodes which failed the schedule phase in the previous attempt,
so it can easily re-place).

This IMHO is has merits over providing the scheduling phase also with a candidate set of nodes:
* how do you decide how many candidate nodes will you provide per request to the scheduler
* Makes the algorithm processing loop simpler. Instead of spitting out a set of candidate
nodes per container request, It just needs to choose the best node. Note that the Algortihm
has at its disposal a full set of nodes to choose from. 

> Add Placement Processor Framework
> ---------------------------------
>                 Key: YARN-7612
>                 URL: https://issues.apache.org/jira/browse/YARN-7612
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: YARN-7612-YARN-6592.001.patch, YARN-7612-YARN-6592.002.patch, YARN-7612-YARN-6592.003.patch,
YARN-7612-YARN-6592.004.patch, YARN-7612-YARN-6592.005.patch, YARN-7612-YARN-6592.006.patch,
YARN-7612-YARN-6592.007.patch, YARN-7612-YARN-6592.008.patch, YARN-7612-v2.wip.patch, YARN-7612.wip.patch
> This introduces a Placement Processor and a Planning algorithm framework to handle placement
constraints and scheduling requests from an app and places them on nodes.
> The actual planning algorithm(s) will be handled in a YARN-7613.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message