hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3306) [Umbrella] Proposing per-queue Policy driven scheduling in YARN
Date Fri, 13 Mar 2015 17:04:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360652#comment-14360652

Craig Welch commented on YARN-3306:

Thanks for your thoughts, [~kasha]

The immediate proposal is to begin adding new functionality in a fashion which can be easily
shared across scheduler implementations and mixed together in a single cluster.  The first
case is to support additional container assignment and preemption types to fifo for applications
in the capacity scheduler and potentially the fair scheduler using the same code, but this
is expected to be expanded to cover queue relationships and potentially other behaviors (limits,
etc) over time.

The hope is that this allows us to iterate toward a state where the various behaviors of the
schedulers can be mixed, matched, and shared across implementations rather than having to
try and accomplish this all in one go, and allows us to achieve the benefit of mixing and
matching some of the features earlier/along the way.

I suspect that at some point we'll hit a critical mass where enough of the functionality has
been extracted to sharable components and where we've been able to establish an understanding
of how these can be made to compose well, and then we'll take that as an inflection point
and go down the path you are suggesting, introduce a new scheduler to house the policies and
in that way complete the picture, deprecating the others.  That's by no means the only possible
conclusion, but it seems to be a good and/or  likely one.

> [Umbrella] Proposing per-queue Policy driven scheduling in YARN
> ---------------------------------------------------------------
>                 Key: YARN-3306
>                 URL: https://issues.apache.org/jira/browse/YARN-3306
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: PerQueuePolicydrivenschedulinginYARN.pdf
> Scheduling layout in Apache Hadoop YARN today is very coarse grained. This proposal aims
at converting today’s rigid scheduling in YARN to a per­-queue policy driven architecture.
> We propose the creation of a c​ommon policy framework​ and implement a​common set
of policies​ that administrators can pick and chose per queue
>  - Make scheduling policies configurable per queue
>  - Initially, we limit ourselves to a new type of scheduling policy that determines the
ordering of applications within the l​eaf ­queue
>  - In the near future, we will also pursue parent­ queue level policies and potential
algorithm reuse through a separate type of policies that control resource limits per queue,
user, application etc.

This message was sent by Atlassian JIRA

View raw message