hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hung <jyhung2...@gmail.com>
Subject [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2
Date Mon, 02 Oct 2017 18:09:13 GMT
Hi all,

>From discussion at [1], I'd like to start a vote to merge feature branch
YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
Monday Oct 9 at 11:00AM PDT.

This branch adds a framework to the scheduler to allow scheduler
configuration mutation on the fly, including a REST and CLI interface, and
an interface for the scheduler configuration backing store. Currently the
capacity scheduler implements this framework.

Umbrella is here (YARN-5734
<https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here (
YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
tasks for this feature are committed. Since this feature changes RM only,
we have tested this on a local RM setup with a suite of configuration
changes with no issue so far.

Key points:
- The feature is turned off by default, and must be explicitly configured
to turn on. When turned off, the behavior reverts back to the original file
based mechanism for changing scheduler configuration (i.e. yarn rmadmin
-refreshQueues).
- The framework was designed in a way to be extendable to other schedulers
(most notably FairScheduler).
- A pluggable ACL policy (YARN-5949
<https://issues.apache.org/jira/browse/YARN-5949>) allows admins
fine-grained control for who can change what configurations.
- The configuration storage backend is also pluggable. Currently an
in-memory, leveldb, and zookeeper implementation are supported.

There were 15 subtasks completed for this feature.

Huge thanks to everyone who helped with reviews, commits, guidance, and
technical discussion/design, including Carlo Curino, Xuan Gong, Subru
Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.

[1]
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E

Jonathan Hung

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message