hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk
Date Thu, 17 Aug 2017 20:28:27 GMT
Hi all,

I want to hear your thoughts of merging YARN resource profile branch into
trunk in the next few weeks. The goal is to get it in for Hadoop 3.0 beta1.

*Regarding to testing:*
We did extensive tests for the feature in the last several months.
Comparing to latest trunk.
- For SLS benchmark: We didn't see observable performance gap from
simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
containers allocated per second.
- For microbenchmark: We use performance test cases added by YARN-6775, it
shows around 5% performance regression comparing to trunk.

*Regarding to API stability: *
Most new added @Public APIs are @Unstable (We're going to convert some new
added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we want
to get this included by beta1 so we get some feedbacks before declaring
stable API.

There're few pending cleanups under YARN-3926 umbrella JIRA. Besides these
cleanups, this feature works from end-to-end, we will do another iteration
of end-to-end tests after cleanup patches got committed.

We would love to get your thoughts before opening a voting thread.

Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Varun
Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
Karthik Kambatla, Jason Lowe, Arun Suresh.

Wangda Tan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message