hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7457) Delay scheduling should be an individual policy instead of part of scheduler implementation
Date Thu, 14 Dec 2017 02:16:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290233#comment-16290233

Wangda Tan commented on YARN-7457:

Reassigned to [~Tao Yang], 

Thanks Tao for sharing this design doc. Comments regarding the design doc: 

1) Overall I think the separate request to update delay scheduling is overkill. We should
have a way to tell YARN how to choose delay scheduling policy as well as delay parameters.
Instead of changing protocol, I suggest initially we can put it under ApplicationSubmissionContext#getAMContainerSpec#environments.
With this, we don’t have to change protocols and all existing YARN apps can use it without
editing code. 

2) I’m doubt if it is a valid case that a single YARN app wants to choose different delay
policies for different SchedulerRequestKey. If it is a non-requirement, #1 should be enough
in the short term until we figure out solid requirements.

3) The original purpose of the JIRA is to make following logics separate from main scheduler
* AppSchedulingInfo#canDelayTo
* AppPlacementAllocator#canDelayTo
* And all delay-scheduling related logics (such as RegularContainerAllocator#canAssign) 
To clarify, this doesn’t necessarily mean we should move these logics to a separate service.
In my opinion, applications should determine how to do delay scheduling by themselves. So
to me a proper place to handle this logic is inside AppPlacementAllocator, but feel free to
suggest if you have any other thoughts.

> Delay scheduling should be an individual policy instead of part of scheduler implementation
> -------------------------------------------------------------------------------------------
>                 Key: YARN-7457
>                 URL: https://issues.apache.org/jira/browse/YARN-7457
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Tao Yang
> Currently, different schedulers have slightly different delay scheduling implementations.
Ideally we should make delay scheduling independent from scheduler implementation. Benefits
of doing this:
> 1) Applications can choose which delay scheduling policy to use, it could be time-based
/ missed-opportunistic-based or whatever new delay scheduling policy supported by the cluster.
Now it is global config of scheduler.
> 2) Make scheduler implementations simpler and reusable.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message