hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil G <sun...@apache.org>
Subject Re: [VOTE] Merge YARN-3926 (resource profile) to trunk
Date Thu, 24 Aug 2017 17:54:14 GMT
Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks who
helped in getting this feature in this level.

Starting with my +1 (binding).


# Tested a 5 node cluster with resource profiles enabled/disabled (feature
is disabled by default)

# All apis added are marked as Unstable/Evolving (very few)

# There is no compatibility break with older versions (we have added UT
cases also to ensure same)

# Performance tests were done using SLS and also with some tight loops unit
tests. There is no much regression with current trunk.

# Latest jenkins +1 on YARN-7013 for whole branch code.

# Verified old RM UI and new YARN UI (newly added resources could be seen
easily)


Once again thanks all the folks who helped in getting this feature. Kudos!


Thanks

- Sunil


On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan <wheeleast@gmail.com> wrote:

>  Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-3926 (Resource profile) to trunk. The vote will run for
> 7 days and will end August 30 10:00 AM PDT.
>
> Briefly, YARN-3926 can extend resource model of YARN to support resource
> types other than CPU and memory, so it will be a cornerstone of features
> like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA
> support (YARN-5983), network IO scheduling/isolation (YARN-2140). In
> addition to that, YARN-3926 allows admin to preconfigure resource profiles
> in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB
> disk>, so applications can request "m3.large" profile instead of specifying
> all resource types’s values.
>
> There are 32 subtasks that were completed as part of this effort.
>
> This feature needs to be explicitly turned on before use. We paid close
> attention to compatibility, performance, and scalability of this feature,
> mentioned in [1], we didn't see observable performance regression in large
> scale SLS (scheduler load simulator) executions and saw less than 5%
> performance regression by using micro benchmark added by YARN-6775.
>
> This feature works from end-to-end (including UI/CLI/application/server),
> we have setup a cluster with this feature turned on runs for several weeks,
> we didn't see any issues by far.
>
> Merge JIRA: YARN-7013 (Jenkins gave +1 already).
> Documentation: YARN-7056
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort including design discussion/development/reviews, etc.: Varun
> Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> Karthik Kambatla, Jason Lowe, Arun Suresh.
>
> Regards,
> Wangda Tan
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%3DBbp4G8inQZmaMg%40mail.gmail.com%3E
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message