hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3806) Proposal of Generic Scheduling Framework for YARN
Date Tue, 16 Jun 2015 21:39:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588827#comment-14588827
] 

Wangda Tan commented on YARN-3806:
----------------------------------

Hi [~wshao],
Thanks for providing thoughts about this, I took a quick look at attached design doc, some
comments (Please correct me if I missed anything).

The JIRA wants to tackle following issues
# Pluggable preemption policy
# Be able to add other allocation policies
# Application level configuration 
# Decouple application / nodes from scheduler

#1/#2, should be already covered by YARN-3306. It doesn't include detailed ParentQueue policy
in design doc, but we plan to extend it to ParentQueue as mentioned in YARN-3306. And I found
"preemptResource" and "acquireResource" in your design are very close to what we have in CS.
If you have time, could you take a look at YARN-3318 (which is already committed), ordering
policy is part of queue policy, is that what you were trying to do?

#3, I'm not sure if it is a valid usecase, I can understand admin can set "maximum-limit"
of application, but setting minimum-share by app sounds not a fair allocation.

#4, Now we have an common abstraction of application (SchedulerApplicationAttempt) and node
(SchedulerNode) for different scheduler implementations. Are you suggesting eliminate scheduler-specific
implementation such as FiCaSchedulerApp/FiCaSchedulerNode? I think that might be problematic,
you can think the app is also a pluggable implementation. For instance, FS/CS have different
logic in app level, such as: how to do delayed scheduling, limits, etc.

To details in the design doc:
- Each LeafQueue can run at most one app, that seems not like a "queue". This is very restrictive,
for example, if there's a requirement needs 1k app running at the same time, do you need to
configure 1k LeafQueues? And how to choose where to submit application?
- YARN-2986 is trying to create a unified view of scheduler configuration, is there any overlap
between YARN-2986 and configuration model mentioned in your doc?

Thoughts?

> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>
>                 Key: YARN-3806
>                 URL: https://issues.apache.org/jira/browse/YARN-3806
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf, ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
>
>
> Currently, a typical YARN cluster runs many different kinds of applications: production
applications, ad hoc user applications, long running services and so on. Different YARN scheduling
policies may be suitable for different applications. For example, capacity scheduling can
manage production applications well since application can get guaranteed resource share, fair
scheduling can manage ad hoc user applications well since it can enforce fairness among users.
However, current YARN scheduling framework doesn’t have a mechanism for multiple scheduling
policies work hierarchically in one cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and proposed
a per-queue policy driven framework. In detail, it supported different scheduling policies
for leaf queues. However, support of different scheduling policies for upper level queues
is not seriously considered yet. 
> A generic scheduling framework is proposed here to address these limitations. It supports
different policies for any queue consistently. The proposal tries to solve many other issues
in current YARN scheduling framework as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message