aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Santhosh Kumar Shanmugham <santhoshkuma...@gmail.com>
Subject Re: Review Request 55284: Benchmarking
Date Mon, 09 Jan 2017 23:28:32 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55284/#review161001
-----------------------------------------------------------


Ship it!





src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java (line 99)
<https://reviews.apache.org/r/55284/#comment232255>

    Increase this to 150k, 200k?


- Santhosh Kumar Shanmugham


On Jan. 8, 2017, 5:04 p.m., Reza Motamedi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55284/
> -----------------------------------------------------------
> 
> (Updated Jan. 8, 2017, 5:04 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> # Adding SLA benchmarks benchmarks 
> This RB is related to [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854).

> The goal is to add JMH benchmarks for SlaCalculation and then other metric calculators.

> 
> I am not sure if the results are very resonable. For instance see that the ops/sec is
54.701±19.743, which seems to have a high variance.
> 
> 
> Diffs
> -----
> 
>   src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 5d8d5bd8f705770979f284d26d2e932aabe707e5

> 
> Diff: https://reviews.apache.org/r/55284/diff/
> 
> 
> Testing
> -------
> 
> Following is the benchmarks collectcted:
> 
> # rmotamedi@tw-mbp-rmotamedi:~/workspace/aurora on git:jmh-sla: [14:39:56]
> > ./gradlew jmh -Pbenchmarks='SlaAlgorithmBenchmark'
> 
> 
> the content of <./aurora/dist/reports/jmh/human.txt>
> 
> ```
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 100)
> 
> # Run progress: 0.00% complete, ETA 00:05:15
> # Fork: 1 of 1
> # Warmup Iteration   1: 127519.626 ops/s
> Iteration   1: 152258.072 ops/s
> Iteration   2: 170404.745 ops/s
> Iteration   3: 138346.324 ops/s
> Iteration   4: 150597.403 ops/s
> Iteration   5: 169640.531 ops/s
> 
> 
> Result "measureMTTA":
>   156249.415 ±(99.9%) 52660.004 ops/s [Average]
>   (min, avg, max) = (138346.324, 156249.415, 170404.745), stdev = 13675.636
>   CI (99.9%): [103589.411, 208909.419] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 1000)
> 
> # Run progress: 11.11% complete, ETA 00:04:45
> # Fork: 1 of 1
> # Warmup Iteration   1: 4329.734 ops/s
> Iteration   1: 4115.153 ops/s
> Iteration   2: 3873.048 ops/s
> Iteration   3: 3479.972 ops/s
> Iteration   4: 3225.995 ops/s
> Iteration   5: 3002.072 ops/s
> 
> 
> Result "measureMTTA":
>   3539.248 ±(99.9%) 1757.523 ops/s [Average]
>   (min, avg, max) = (3002.072, 3539.248, 4115.153), stdev = 456.423
>   CI (99.9%): [1781.725, 5296.771] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 10000)
> 
> # Run progress: 22.22% complete, ETA 00:04:10
> # Fork: 1 of 1
> # Warmup Iteration   1: 44.147 ops/s
> Iteration   1: 46.116 ops/s
> Iteration   2: 54.255 ops/s
> Iteration   3: 57.967 ops/s
> Iteration   4: 56.174 ops/s
> Iteration   5: 58.992 ops/s
> 
> 
> Result "measureMTTA":
>   54.701 ±(99.9%) 19.743 ops/s [Average]
>   (min, avg, max) = (46.116, 54.701, 58.992), stdev = 5.127
>   CI (99.9%): [34.957, 74.444] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 100)
> 
> # Run progress: 33.33% complete, ETA 00:03:38
> # Fork: 1 of 1
> # Warmup Iteration   1: 106633.191 ops/s
> Iteration   1: 136861.727 ops/s
> Iteration   2: 120490.231 ops/s
> Iteration   3: 106383.718 ops/s
> Iteration   4: 126247.774 ops/s
> Iteration   5: 131648.970 ops/s
> 
> 
> Result "measureMTTR":
>   124326.484 ±(99.9%) 45198.021 ops/s [Average]
>   (min, avg, max) = (106383.718, 124326.484, 136861.727), stdev = 11737.783
>   CI (99.9%): [79128.463, 169524.505] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 1000)
> 
> # Run progress: 44.44% complete, ETA 00:03:01
> # Fork: 1 of 1
> # Warmup Iteration   1: 4012.815 ops/s
> Iteration   1: 4396.495 ops/s
> Iteration   2: 4086.905 ops/s
> Iteration   3: 4024.187 ops/s
> Iteration   4: 3809.202 ops/s
> Iteration   5: 3551.905 ops/s
> 
> 
> Result "measureMTTR":
>   3973.739 ±(99.9%) 1216.296 ops/s [Average]
>   (min, avg, max) = (3551.905, 3973.739, 4396.495), stdev = 315.868
>   CI (99.9%): [2757.443, 5190.034] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 10000)
> 
> # Run progress: 55.56% complete, ETA 00:02:25
> # Fork: 1 of 1
> # Warmup Iteration   1: 50.933 ops/s
> Iteration   1: 57.153 ops/s
> Iteration   2: 58.741 ops/s
> Iteration   3: 56.398 ops/s
> Iteration   4: 55.078 ops/s
> Iteration   5: 58.142 ops/s
> 
> 
> Result "measureMTTR":
>   57.102 ±(99.9%) 5.567 ops/s [Average]
>   (min, avg, max) = (55.078, 57.102, 58.741), stdev = 1.446
>   CI (99.9%): [51.535, 62.670] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 100)
> 
> # Run progress: 66.67% complete, ETA 00:01:49
> # Fork: 1 of 1
> # Warmup Iteration   1: 107724.308 ops/s
> Iteration   1: 121787.913 ops/s
> Iteration   2: 119594.939 ops/s
> Iteration   3: 128441.840 ops/s
> Iteration   4: 134176.538 ops/s
> Iteration   5: 144275.262 ops/s
> 
> 
> Result "measureMTTS":
>   129655.298 ±(99.9%) 38446.901 ops/s [Average]
>   (min, avg, max) = (119594.939, 129655.298, 144275.262), stdev = 9984.538
>   CI (99.9%): [91208.397, 168102.199] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 1000)
> 
> # Run progress: 77.78% complete, ETA 00:01:12
> # Fork: 1 of 1
> # Warmup Iteration   1: 3738.823 ops/s
> Iteration   1: 4026.955 ops/s
> Iteration   2: 3070.155 ops/s
> Iteration   3: 2706.577 ops/s
> Iteration   4: 3510.103 ops/s
> Iteration   5: 4063.217 ops/s
> 
> 
> Result "measureMTTS":
>   3475.401 ±(99.9%) 2283.158 ops/s [Average]
>   (min, avg, max) = (2706.577, 3475.401, 4063.217), stdev = 592.929
>   CI (99.9%): [1192.244, 5758.559] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 98 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 10000)
> 
> # Run progress: 88.89% complete, ETA 00:00:36
> # Fork: 1 of 1
> # Warmup Iteration   1: 56.489 ops/s
> Iteration   1: 59.288 ops/s
> Iteration   2: 59.160 ops/s
> Iteration   3: 56.142 ops/s
> Iteration   4: 53.336 ops/s
> Iteration   5: 53.147 ops/s
> 
> 
> Result "measureMTTS":
>   56.215 ±(99.9%) 11.524 ops/s [Average]
>   (min, avg, max) = (53.147, 56.215, 59.288), stdev = 2.993
>   CI (99.9%): [44.691, 67.738] (assumes normal distribution)
> 
> 
> # Run complete. Total time: 00:05:27
> 
> Benchmark                                        (numTasks)   Mode  Cnt       Score 
     Error  Units
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA         100  thrpt    5  156249.415 ±
52660.004  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA        1000  thrpt    5    3539.248 ±
 1757.523  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA       10000  thrpt    5      54.701 ±
   19.743  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR         100  thrpt    5  124326.484 ±
45198.021  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR        1000  thrpt    5    3973.739 ±
 1216.296  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR       10000  thrpt    5      57.102 ±
    5.567  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS         100  thrpt    5  129655.298 ±
38446.901  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS        1000  thrpt    5    3475.401 ±
 2283.158  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS       10000  thrpt    5      56.215 ±
   11.524  ops/s
> 
> Benchmark result is saved to dist/reports/jmh/results.txt
> 
> ```
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message