hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3306) [Umbrella] Proposing per-queue Policy driven scheduling in YARN
Date Wed, 13 May 2015 22:11:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542829#comment-14542829

Craig Welch commented on YARN-3306:

bq. Integration points being narrow is probably more a reason to work on a branch

I don't see why - it seems to me to suggest the opposite, that we are able to achieve isolation
of the functionality in the main codebase without risk

bq. The code looks quite isolated

Another reason we don't need to work in a branch

We're trying to approach this iteratively, building specific, narrow functionalities to completion
and then making them available for use and feedback, this will be difficult if it's all isolated
away in a branch.  The approach so far works well for that process - much better than doing
all the work in isolation and then bringing a much larger change into the main codebase all
at once.  

As far as I can tell, separating this out into a branch is a net negative- there is overhead
to doing so and it runs contrary to the iterative approach we're trying to take, without providing
any clear benefit.

> [Umbrella] Proposing per-queue Policy driven scheduling in YARN
> ---------------------------------------------------------------
>                 Key: YARN-3306
>                 URL: https://issues.apache.org/jira/browse/YARN-3306
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: PerQueuePolicydrivenschedulinginYARN.pdf
> Scheduling layout in Apache Hadoop YARN today is very coarse grained. This proposal aims
at converting today’s rigid scheduling in YARN to a per­-queue policy driven architecture.
> We propose the creation of a c​ommon policy framework​ and implement a​common set
of policies​ that administrators can pick and chose per queue
>  - Make scheduling policies configurable per queue
>  - Initially, we limit ourselves to a new type of scheduling policy that determines the
ordering of applications within the l​eaf ­queue
>  - In the near future, we will also pursue parent­ queue level policies and potential
algorithm reuse through a separate type of policies that control resource limits per queue,
user, application etc.

This message was sent by Atlassian JIRA

View raw message