hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil G <sun...@apache.org>
Subject Re: [VOTE] Merge YARN-3926 (resource profile) to trunk
Date Sat, 26 Aug 2017 15:48:01 GMT
Thanks Arun for checking the feature.

* can you folks point me to any test application / framework or has this
been integrated with MapReduce
Currently this feature is integrated with Distributed Shell.
Reference: YARN-5588
This is not yet integrated with MapReduce. This work is ongoing
in YARN-6504.

*  Can you maybe comment a bit on the type of scale testing done ?
We have done scale testing by using SLS with this feature turned off and
also turned on with only Memory and VCores. This performance was on par
with trunk with a variance of ~2%. I will let Wangda to add more color here
with data.

*  Is there a plan to merge this with branch-2 ?
We had a discussion with few folks here in Bangalore from MS and Huawei.
And will be looking into same as this branch is merged in trunk.

- Sunil



On Sat, Aug 26, 2017 at 9:47 AM Arun Suresh <arun.suresh@gmail.com> wrote:

> Really looking forward to getting this in.
>
> Couple of questions:
> * Can you maybe comment a bit on the type of scale testing done ?
> Specifically, the number of resources tested with and any point where it is
> discovered that performance might take a hit. Also, given that we do not
> have AM's that currently use this feature, can you folks point me to any
> test application / framework or has this been integrated with MapReduce ?
> * Is there a plan to merge this with branch-2 ? - Since we would like to
> see this in 2.9.0 as well.
>
> Just to clarify, I am a +1 for merging, irrespective of the above - given
> that this is an opt-in feature after all. I am just eager to start using it
> :)
>
> Cheers
> -Arun
>
>
> On Thu, Aug 24, 2017 at 10:54 AM, Sunil G <sunilg@apache.org> wrote:
>
>> Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks
>> who
>> helped in getting this feature in this level.
>>
>> Starting with my +1 (binding).
>>
>>
>> # Tested a 5 node cluster with resource profiles enabled/disabled (feature
>> is disabled by default)
>>
>> # All apis added are marked as Unstable/Evolving (very few)
>>
>> # There is no compatibility break with older versions (we have added UT
>> cases also to ensure same)
>>
>> # Performance tests were done using SLS and also with some tight loops
>> unit
>> tests. There is no much regression with current trunk.
>>
>> # Latest jenkins +1 on YARN-7013 for whole branch code.
>>
>> # Verified old RM UI and new YARN UI (newly added resources could be seen
>> easily)
>>
>>
>> Once again thanks all the folks who helped in getting this feature. Kudos!
>>
>>
>> Thanks
>>
>> - Sunil
>>
>>
>> On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan <wheeleast@gmail.com> wrote:
>>
>> >  Hi folks,
>> >
>> > Per earlier discussion [1], I'd like to start a formal vote to merge
>> > feature branch YARN-3926 (Resource profile) to trunk. The vote will run
>> for
>> > 7 days and will end August 30 10:00 AM PDT.
>> >
>> > Briefly, YARN-3926 can extend resource model of YARN to support resource
>> > types other than CPU and memory, so it will be a cornerstone of features
>> > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139),
>> FPGA
>> > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In
>> > addition to that, YARN-3926 allows admin to preconfigure resource
>> profiles
>> > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64
>> GB
>> > disk>, so applications can request "m3.large" profile instead of
>> specifying
>> > all resource types’s values.
>> >
>> > There are 32 subtasks that were completed as part of this effort.
>> >
>> > This feature needs to be explicitly turned on before use. We paid close
>> > attention to compatibility, performance, and scalability of this
>> feature,
>> > mentioned in [1], we didn't see observable performance regression in
>> large
>> > scale SLS (scheduler load simulator) executions and saw less than 5%
>> > performance regression by using micro benchmark added by YARN-6775.
>> >
>> > This feature works from end-to-end (including
>> UI/CLI/application/server),
>> > we have setup a cluster with this feature turned on runs for several
>> weeks,
>> > we didn't see any issues by far.
>> >
>> > Merge JIRA: YARN-7013 (Jenkins gave +1 already).
>> > Documentation: YARN-7056
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this effort including design discussion/development/reviews, etc.: Varun
>> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
>> > Karthik Kambatla, Jason Lowe, Arun Suresh.
>> >
>> > Regards,
>> > Wangda Tan
>> >
>> > [1]
>> >
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%3DBbp4G8inQZmaMg%40mail.gmail.com%3E
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message