hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@apache.org>
Subject Re: [DISCUSS] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk
Date Wed, 29 Nov 2017 13:23:33 GMT
+1, thanks Sunil for working on this feature!

-Rohith Sharma K S

On 24 November 2017 at 23:19, Sunil G <sunilg@apache.org> wrote:

> Hi All,
>
> We would like to bring up the discussion of merging “absolute min/max
> resources support in capacity scheduler” branch (YARN-5881) [2] into trunk
> in a few weeks. The goal is to get it in for Hadoop 3.1.
>
> *Major work happened in this branch*
>
>    - YARN-6471. Support to add min/max resource configuration for a queue
>    - YARN-7332. Compute effectiveCapacity per each resource vector
>    - YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to
>    handle absolute resources.
>
> *Regarding design details*
>
> Please refer [1] for detailed design document.
>
> *Regarding to testing:*
>
> We did extensive tests for the feature in the last couple of months.
> Comparing to latest trunk.
>
> - For SLS benchmark: We didn't see observable performance gap from
> simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
> containers allocated per second.
>
> - For microbenchmark: We use performance test cases added by YARN 6775, it
> did not show much performance regression comparing to trunk.
>
> *YARN-5881* <https://issues.apache.org/jira/browse/YARN-5881>
>
> #ResourceTypes = 2. Avg of fastest 20: 55294.52
> #ResourceTypes = 2. Avg of fastest 20: 55401.66
>
> *trunk*
> #ResourceTypes = 2. Avg of fastest 20: 55865.92
> #ResourceTypes = 2. Avg of fastest 20: 55096.418
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we are running this in our
> development cluster for last couple of months and undergone good amount of
> testing. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Wangda
> Tan, Vinod Kumar Vavilappali, Rohith Sharma K S.
>
> [1] :
> https://issues.apache.org/jira/secure/attachment/
> 12855984/YARN-5881.Support.Absolute.Min.Max.Resource.In.
> Capacity.Scheduler.design-doc.v1.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-5881
>
> [3] : https://issues.apache.org/jira/browse/YARN-7533
>
> [4] : https://issues.apache.org/jira/browse/YARN-7510
>
> Thanks,
>
> Sunil G and Wangda Tan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message