aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxim Khutornenko" <ma...@apache.org>
Subject Re: Review Request 32802: Add a benchmark for getRoleSummary.
Date Fri, 03 Apr 2015 18:03:56 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/#review78800
-----------------------------------------------------------

Ship it!



src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java
<https://reviews.apache.org/r/32802/#comment127824>

    Nice! I could not make this work through annotations in earlier versions of jmh/gradle
plugin.



src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java
<https://reviews.apache.org/r/32802/#comment127844>

    Perhaps just hardcode this in Tasks.Builder() instead and drop the builder method? I don't
see why we would ever want to change this value.


- Maxim Khutornenko


On April 3, 2015, 2:19 a.m., Bill Farner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32802/
> -----------------------------------------------------------
> 
> (Updated April 3, 2015, 2:19 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> The first of several benchmarks to exercise the thrift API.  The real motivation here
is to have performance data to assess the impact of migrating the task store to H2.  I'm targeting
the `getRoleSummary` RPC specifically here as i consider it the likely candidate to suffer.
> 
> One 'trick' i employed here was JSON for input parameters.  I took this approach since
i really wanted to explore specific axes of growth without the full outer product of each
individual parameters.
> 
> Some changes along the way:
> - upgraded to jmh 1.7.1
> - moved some JMH run configuration parameters out of the gradle config and into benchmark
classes.  This allows us to define run configurations for each microbenchmark
> - changed the benchmark mode for existing and new tests to throughput.  We had several
benchmarks whose iteration time was measuring in hundreds of microseconds, which is too high
resolution to capture performance variation.  By running iterations for a set amount of time
and counting iterations accomplished, we can capture more reliable data.
> 
> 
> Diffs
> -----
> 
>   build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb

>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 491c6871ba7a4c76a4eb084afb2627a3f93db8df

>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java c65e0218211b797ad50e2ac62e71136f967e5f5d

> 
> Diff: https://reviews.apache.org/r/32802/diff/
> 
> 
> Testing
> -------
> 
> FYI numbers and stddev are not useful in the data captured below, as my machine was doing
other intensive things while running this.  Output is more useful as an example of how the
data and run configurations are presented.
> 
> ```
> # Run complete. Total time: 00:18:51
> 
> Benchmark                                                                      (testConfiguration)
  Mode  Cnt       Score       Error  Units
> SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark                   
          N/A  thrpt   10  499824.589 ± 86328.152  ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark          
          N/A  thrpt   10   40482.762 ±  5394.306  ops/s
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark        
          N/A  thrpt   10    2933.958 ±   577.571  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
          N/A  thrpt   10     220.079 ±    28.943  ops/s
> SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark        
          N/A  thrpt   10   41948.230 ±  1906.106  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     
 {"roles": 1}  thrpt    5    1978.614 ±   736.704  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     
{"roles": 10}  thrpt    5     143.465 ±    85.579  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     {"roles":
100}  thrpt    5       4.003 ±     2.742  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     {"roles":
500}  thrpt    5       1.964 ±     0.569  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     
  {"jobs": 1}  thrpt    5    4073.564 ±  3343.602  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     
 {"jobs": 10}  thrpt    5    4401.662 ±  3943.372  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     
{"jobs": 100}  thrpt    5    4327.726 ±   726.900  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                     
{"jobs": 500}  thrpt    5    2473.799 ±  5396.188  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                   {"instances":
1}  thrpt    5    7351.543 ± 13368.279  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                  {"instances":
10}  thrpt    5    9496.478 ±  3066.478  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 {"instances":
100}  thrpt    5    4640.480 ±  3307.831  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                {"instances":
1000}  thrpt    5     429.841 ±   191.521  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               {"instances":
10000}  thrpt    5      23.327 ±     7.011  ops/s
> ```
> 
> 
> Thanks,
> 
> Bill Farner
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message