hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7457) Delay scheduling should be an individual policy instead of part of scheduler implementation
Date Fri, 15 Dec 2017 04:10:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292012#comment-16292012

Wangda Tan commented on YARN-7457:

bq. Or you are thinking of ditching the missed opportunities option?
I would prefer to ditch missed opportunities option. 

bq. New apps will be using the delayed-or constraint from YARN-6592, right?

bq. With the string to string map, you are suggesting to add something like "delay_to_rack"
-> "10 sec" as a hint?

bq. If these are legacy apps, they will still have to be modified to use this new hint.
Not really, as I mentioned above, there's one way to let legacy apps to use the new hint without
modifying the code (idea credit to [~sunilg]). 
bq. ... and it will copy AMSpec#environment if the scheduling hint map is absent. ...

bq. That said, do you expect this to get all the way to the scheduler, or at some point to
automatically translate these to SchedulingRequests? I think it will get messy to have two
code paths for that.
I haven't thought about the implementation of this yet, ideally delay policy implementation
should not depend on SchedulingRequest, it only takes input from SchedulingRequest (or hint
environments). We can figure out impl details while working on the logics.

> Delay scheduling should be an individual policy instead of part of scheduler implementation
> -------------------------------------------------------------------------------------------
>                 Key: YARN-7457
>                 URL: https://issues.apache.org/jira/browse/YARN-7457
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Tao Yang
> Currently, different schedulers have slightly different delay scheduling implementations.
Ideally we should make delay scheduling independent from scheduler implementation. Benefits
of doing this:
> 1) Applications can choose which delay scheduling policy to use, it could be time-based
/ missed-opportunistic-based or whatever new delay scheduling policy supported by the cluster.
Now it is global config of scheduler.
> 2) Make scheduler implementations simpler and reusable.
> h2. {color:red}Running design doc: https://docs.google.com/document/d/1rY-CJPLbGk3Xj_8sxre61y2YkHJFK8oqKOshro1ZY3A/edit#heading=h.xnzvh9nn283a{color}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message