hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Suresh <arun.sur...@gmail.com>
Subject Re: [VOTE] Merge YARN-3926 (resource profile) to trunk
Date Sat, 26 Aug 2017 04:17:03 GMT
Really looking forward to getting this in.

Couple of questions:
* Can you maybe comment a bit on the type of scale testing done ?
Specifically, the number of resources tested with and any point where it is
discovered that performance might take a hit. Also, given that we do not
have AM's that currently use this feature, can you folks point me to any
test application / framework or has this been integrated with MapReduce ?
* Is there a plan to merge this with branch-2 ? - Since we would like to
see this in 2.9.0 as well.

Just to clarify, I am a +1 for merging, irrespective of the above - given
that this is an opt-in feature after all. I am just eager to start using it
:)

Cheers
-Arun


On Thu, Aug 24, 2017 at 10:54 AM, Sunil G <sunilg@apache.org> wrote:

> Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks who
> helped in getting this feature in this level.
>
> Starting with my +1 (binding).
>
>
> # Tested a 5 node cluster with resource profiles enabled/disabled (feature
> is disabled by default)
>
> # All apis added are marked as Unstable/Evolving (very few)
>
> # There is no compatibility break with older versions (we have added UT
> cases also to ensure same)
>
> # Performance tests were done using SLS and also with some tight loops unit
> tests. There is no much regression with current trunk.
>
> # Latest jenkins +1 on YARN-7013 for whole branch code.
>
> # Verified old RM UI and new YARN UI (newly added resources could be seen
> easily)
>
>
> Once again thanks all the folks who helped in getting this feature. Kudos!
>
>
> Thanks
>
> - Sunil
>
>
> On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan <wheeleast@gmail.com> wrote:
>
> >  Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-3926 (Resource profile) to trunk. The vote will run
> for
> > 7 days and will end August 30 10:00 AM PDT.
> >
> > Briefly, YARN-3926 can extend resource model of YARN to support resource
> > types other than CPU and memory, so it will be a cornerstone of features
> > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA
> > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In
> > addition to that, YARN-3926 allows admin to preconfigure resource
> profiles
> > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB
> > disk>, so applications can request "m3.large" profile instead of
> specifying
> > all resource types’s values.
> >
> > There are 32 subtasks that were completed as part of this effort.
> >
> > This feature needs to be explicitly turned on before use. We paid close
> > attention to compatibility, performance, and scalability of this feature,
> > mentioned in [1], we didn't see observable performance regression in
> large
> > scale SLS (scheduler load simulator) executions and saw less than 5%
> > performance regression by using micro benchmark added by YARN-6775.
> >
> > This feature works from end-to-end (including UI/CLI/application/server),
> > we have setup a cluster with this feature turned on runs for several
> weeks,
> > we didn't see any issues by far.
> >
> > Merge JIRA: YARN-7013 (Jenkins gave +1 already).
> > Documentation: YARN-7056
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort including design discussion/development/reviews, etc.: Varun
> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> > Karthik Kambatla, Jason Lowe, Arun Suresh.
> >
> > Regards,
> > Wangda Tan
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/
> 201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%
> 3DBbp4G8inQZmaMg%40mail.gmail.com%3E
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message