hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3806) Proposal of Generic Scheduling Framework for YARN
Date Tue, 16 Jun 2015 22:56:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588955#comment-14588955

Wei Shao commented on YARN-3806:

Hi Wangda Tan,

Thanks for reading the proposal! Some quick replies.

1. The queue which the applications submit to will create leaf queue for each accepted application
in runtime, application won't submit to leaf queue directly. With the concept of 'single application
queue', scheduling framework can manage queues and applications (since application is also
a queue) in tree consistently. The scheduler even doesn't need know the concept of application.
So the proposed preemption model (support preemption among applications in same queue), resource
allocation model, and configuration model (support application level configuration) can be
implemented consistently. This is one of the design difference between YARN-3306 and this

2. Application level configuration. Yes, application specific minimalShare for fair scheduling
isn't useful in terms of fairness. However, there are some other cases it might be useful.
   a. In configuration section for the queue with fair scheduling, there could be a template
configuration for all single application queues it will create in runtime, and minimalThreshold
can be specified there. Then minimalShare for each application is minimalThreshold*fairShare
and updated at runtime. Preemption model can preempt resources between applications according
to minimalShare while respecting fairness.
   b. For the queue with capacity scheduling or scheduling policy proposed in YARN-3807, each
application certainly can have its own capacity to meeting SLA.
By having 'single application queue', all application configurations can be respected consistently.

Regarding your other comments, I will take a look at the jiras you mentioned and reply later.

> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>                 Key: YARN-3806
>                 URL: https://issues.apache.org/jira/browse/YARN-3806
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf, ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
> Currently, a typical YARN cluster runs many different kinds of applications: production
applications, ad hoc user applications, long running services and so on. Different YARN scheduling
policies may be suitable for different applications. For example, capacity scheduling can
manage production applications well since application can get guaranteed resource share, fair
scheduling can manage ad hoc user applications well since it can enforce fairness among users.
However, current YARN scheduling framework doesn’t have a mechanism for multiple scheduling
policies work hierarchically in one cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and proposed
a per-queue policy driven framework. In detail, it supported different scheduling policies
for leaf queues. However, support of different scheduling policies for upper level queues
is not seriously considered yet. 
> A generic scheduling framework is proposed here to address these limitations. It supports
different policies for any queue consistently. The proposal tries to solve many other issues
in current YARN scheduling framework as well.

This message was sent by Atlassian JIRA

View raw message