Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Wed, 17 Jun 2015 02:32:02 +0000 (UTC)
From: "Wei Shao (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12838037.1434426977000.92779.1434508322355@Atlassian.JIRA>
In-Reply-To: <JIRA.12838037.1434426977000@Atlassian.JIRA>
References: <JIRA.12838037.1434426977000@Atlassian.JIRA>
 <JIRA.12838037.1434426977903@arcas>
Subject: [jira] [Commented] (YARN-3806) Proposal of Generic Scheduling
 Framework for YARN
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/YARN-3806?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14589=
201#comment-14589201 ]=20

Wei Shao commented on YARN-3806:
--------------------------------

Hi Wangda Tan,

Regarding #4 in your comments. (Decouple application / nodes from scheduler=
).

This proposal suggests new object ResourceManager (or we could call it Sche=
dulerNodeManager) to manage SchedulerNodes and handle all events from clust=
ers nodes. SchedulerManager doesn't response to these events directly.
In current implementation of FiCaSchedulerNode, It looks to me container re=
servation feature may don't need to bind with fair/capacity scheduling. Fai=
r/capacity scheduling can use it, but other scheduling policies can choose =
to use it or not as well.

And by single application queue introduced in the proposal, maybe the sched=
uler-specific features of FiCaSchedulerApp can be moved to the implementati=
on of specific scheduler queue, like resource limits. And parent queues and=
 single application queues can implement these features consistently. Also,=
 it looks to me delayed scheduling feature may don't need to bind with fair=
/capacity scheduling, any scheduler can choose to use it or not.

By proposal, in each scheduling cycle, the SchedulerManager reads status of=
 cluster resources from ResourceManager, updates scheduling parameters (fai=
rShare, resource limits and so on) consistently for all queues (application=
 is also queue), and sends resource preemption/allocation events to Schedul=
erApp, SchedulerApp can implement container warning feature in preemptResou=
rce() and delayed scheduling feature in acquireResource(), which are applic=
able for all schedulers. Also, scheduler doesn't specify the resources Sche=
dulerApp can get, SchedulerApp.acquireResource() asks available resources f=
rom ResourceManager directly.

And by proposal, the procedures to update scheduling parameters are scalabl=
e (by parallelism), idempotent, and transactional. See detail in proposal f=
or why these properties can be helpful.

Since both YARN-3306 and this proposal are trying to address similar issues=
, If some ideas in this proposal are useful, maybe efforts can be combined.

Thoughts? Thanks!

> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>
>                 Key: YARN-3806
>                 URL: https://issues.apache.org/jira/browse/YARN-3806
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf=
, ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
>
>
> Currently, a typical YARN cluster runs many different kinds of applicatio=
ns: production applications, ad hoc user applications, long running service=
s and so on. Different YARN scheduling policies may be suitable for differe=
nt applications. For example, capacity scheduling can manage production app=
lications well since application can get guaranteed resource share, fair sc=
heduling can manage ad hoc user applications well since it can enforce fair=
ness among users. However, current YARN scheduling framework doesn=E2=80=99=
t have a mechanism for multiple scheduling policies work hierarchically in =
one cluster.
> YARN-3306 talked about many issues of today=E2=80=99s YARN scheduling fra=
mework, and proposed a per-queue policy driven framework. In detail, it sup=
ported different scheduling policies for leaf queues. However, support of d=
ifferent scheduling policies for upper level queues is not seriously consid=
ered yet.=20
> A generic scheduling framework is proposed here to address these limitati=
ons. It supports different policies for any queue consistently. The proposa=
l tries to solve many other issues in current YARN scheduling framework as =
well.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)