hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3806) Proposal of Generic Scheduling Framework for YARN
Date Tue, 16 Jun 2015 23:42:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589022#comment-14589022
] 

Wei Shao commented on YARN-3806:
--------------------------------

Regarding YARN-2986 about unified view of scheduler configuration. I definitely +1 for having
a single unified configuration file.

In addition, to support dynamic created 'single application queue' proposed here. I suggest
to add following feature as well.

In configuration file, each queue can have a subsection to specify the configuration templates
for dynamic created sub queues. And each configuration template can have an id which applications
can reference to. 
For example, in configuration file, for a queue with capacity scheduling, configuration templates
with different priorities (template id means priority) can be specified. The applications
submitted to the queue can specify different priority. The queue will create leaf queue with
specified configuration for the application.




> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>
>                 Key: YARN-3806
>                 URL: https://issues.apache.org/jira/browse/YARN-3806
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf, ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
>
>
> Currently, a typical YARN cluster runs many different kinds of applications: production
applications, ad hoc user applications, long running services and so on. Different YARN scheduling
policies may be suitable for different applications. For example, capacity scheduling can
manage production applications well since application can get guaranteed resource share, fair
scheduling can manage ad hoc user applications well since it can enforce fairness among users.
However, current YARN scheduling framework doesn’t have a mechanism for multiple scheduling
policies work hierarchically in one cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and proposed
a per-queue policy driven framework. In detail, it supported different scheduling policies
for leaf queues. However, support of different scheduling policies for upper level queues
is not seriously considered yet. 
> A generic scheduling framework is proposed here to address these limitations. It supports
different policies for any queue consistently. The proposal tries to solve many other issues
in current YARN scheduling framework as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message