hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhe Zhang <...@apache.org>
Subject Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2
Date Mon, 02 Oct 2017 18:23:33 GMT
+1 (binding)

LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.

I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.

On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jyhung2357@gmail.com> wrote:

> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message