aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reza Motamedi <reza.motam...@gmail.com>
Subject Review Request 55284: Benchmarking
Date Fri, 06 Jan 2017 23:11:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55284/
-----------------------------------------------------------

Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh Kumar Shanmugham.


Repository: aurora


Description
-------

# Adding SLA benchmarks benchmarks 
This RB is related to [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854). 
The goal is to add JMH benchmarks for SlaCalculation and then other metric calculators. 

I am not sure if the results are very resonable. For instance see that the ops/sec is 54.701±19.743,
which seems to have a high variance.


Diffs
-----

  src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 5d8d5bd8f705770979f284d26d2e932aabe707e5


Diff: https://reviews.apache.org/r/55284/diff/


Testing
-------

Following is the benchmarks collectcted:

# rmotamedi@tw-mbp-rmotamedi:~/workspace/aurora on git:jmh-sla ?? [14:39:56]
? ./gradlew jmh -Pbenchmarks='SlaAlgorithmBenchmark'


the content of <./aurora/dist/reports/jmh/human.txt>

```
# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 100)

# Run progress: 0.00% complete, ETA 00:05:15
# Fork: 1 of 1
# Warmup Iteration   1: 127519.626 ops/s
Iteration   1: 152258.072 ops/s
Iteration   2: 170404.745 ops/s
Iteration   3: 138346.324 ops/s
Iteration   4: 150597.403 ops/s
Iteration   5: 169640.531 ops/s


Result "measureMTTA":
  156249.415 ±(99.9%) 52660.004 ops/s [Average]
  (min, avg, max) = (138346.324, 156249.415, 170404.745), stdev = 13675.636
  CI (99.9%): [103589.411, 208909.419] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 1000)

# Run progress: 11.11% complete, ETA 00:04:45
# Fork: 1 of 1
# Warmup Iteration   1: 4329.734 ops/s
Iteration   1: 4115.153 ops/s
Iteration   2: 3873.048 ops/s
Iteration   3: 3479.972 ops/s
Iteration   4: 3225.995 ops/s
Iteration   5: 3002.072 ops/s


Result "measureMTTA":
  3539.248 ±(99.9%) 1757.523 ops/s [Average]
  (min, avg, max) = (3002.072, 3539.248, 4115.153), stdev = 456.423
  CI (99.9%): [1781.725, 5296.771] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 10000)

# Run progress: 22.22% complete, ETA 00:04:10
# Fork: 1 of 1
# Warmup Iteration   1: 44.147 ops/s
Iteration   1: 46.116 ops/s
Iteration   2: 54.255 ops/s
Iteration   3: 57.967 ops/s
Iteration   4: 56.174 ops/s
Iteration   5: 58.992 ops/s


Result "measureMTTA":
  54.701 ±(99.9%) 19.743 ops/s [Average]
  (min, avg, max) = (46.116, 54.701, 58.992), stdev = 5.127
  CI (99.9%): [34.957, 74.444] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 100)

# Run progress: 33.33% complete, ETA 00:03:38
# Fork: 1 of 1
# Warmup Iteration   1: 106633.191 ops/s
Iteration   1: 136861.727 ops/s
Iteration   2: 120490.231 ops/s
Iteration   3: 106383.718 ops/s
Iteration   4: 126247.774 ops/s
Iteration   5: 131648.970 ops/s


Result "measureMTTR":
  124326.484 ±(99.9%) 45198.021 ops/s [Average]
  (min, avg, max) = (106383.718, 124326.484, 136861.727), stdev = 11737.783
  CI (99.9%): [79128.463, 169524.505] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 1000)

# Run progress: 44.44% complete, ETA 00:03:01
# Fork: 1 of 1
# Warmup Iteration   1: 4012.815 ops/s
Iteration   1: 4396.495 ops/s
Iteration   2: 4086.905 ops/s
Iteration   3: 4024.187 ops/s
Iteration   4: 3809.202 ops/s
Iteration   5: 3551.905 ops/s


Result "measureMTTR":
  3973.739 ±(99.9%) 1216.296 ops/s [Average]
  (min, avg, max) = (3551.905, 3973.739, 4396.495), stdev = 315.868
  CI (99.9%): [2757.443, 5190.034] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 10000)

# Run progress: 55.56% complete, ETA 00:02:25
# Fork: 1 of 1
# Warmup Iteration   1: 50.933 ops/s
Iteration   1: 57.153 ops/s
Iteration   2: 58.741 ops/s
Iteration   3: 56.398 ops/s
Iteration   4: 55.078 ops/s
Iteration   5: 58.142 ops/s


Result "measureMTTR":
  57.102 ±(99.9%) 5.567 ops/s [Average]
  (min, avg, max) = (55.078, 57.102, 58.741), stdev = 1.446
  CI (99.9%): [51.535, 62.670] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 100)

# Run progress: 66.67% complete, ETA 00:01:49
# Fork: 1 of 1
# Warmup Iteration   1: 107724.308 ops/s
Iteration   1: 121787.913 ops/s
Iteration   2: 119594.939 ops/s
Iteration   3: 128441.840 ops/s
Iteration   4: 134176.538 ops/s
Iteration   5: 144275.262 ops/s


Result "measureMTTS":
  129655.298 ±(99.9%) 38446.901 ops/s [Average]
  (min, avg, max) = (119594.939, 129655.298, 144275.262), stdev = 9984.538
  CI (99.9%): [91208.397, 168102.199] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 1000)

# Run progress: 77.78% complete, ETA 00:01:12
# Fork: 1 of 1
# Warmup Iteration   1: 3738.823 ops/s
Iteration   1: 4026.955 ops/s
Iteration   2: 3070.155 ops/s
Iteration   3: 2706.577 ops/s
Iteration   4: 3510.103 ops/s
Iteration   5: 4063.217 ops/s


Result "measureMTTS":
  3475.401 ±(99.9%) 2283.158 ops/s [Average]
  (min, avg, max) = (2706.577, 3475.401, 4063.217), stdev = 592.929
  CI (99.9%): [1192.244, 5758.559] (assumes normal distribution)


# JMH 1.15 (released 98 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 10000)

# Run progress: 88.89% complete, ETA 00:00:36
# Fork: 1 of 1
# Warmup Iteration   1: 56.489 ops/s
Iteration   1: 59.288 ops/s
Iteration   2: 59.160 ops/s
Iteration   3: 56.142 ops/s
Iteration   4: 53.336 ops/s
Iteration   5: 53.147 ops/s


Result "measureMTTS":
  56.215 ±(99.9%) 11.524 ops/s [Average]
  (min, avg, max) = (53.147, 56.215, 59.288), stdev = 2.993
  CI (99.9%): [44.691, 67.738] (assumes normal distribution)


# Run complete. Total time: 00:05:27

Benchmark                                        (numTasks)   Mode  Cnt       Score      
Error  Units
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA         100  thrpt    5  156249.415 ± 52660.004
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA        1000  thrpt    5    3539.248 ±  1757.523
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA       10000  thrpt    5      54.701 ±   
19.743  ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR         100  thrpt    5  124326.484 ± 45198.021
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR        1000  thrpt    5    3973.739 ±  1216.296
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR       10000  thrpt    5      57.102 ±   
 5.567  ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS         100  thrpt    5  129655.298 ± 38446.901
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS        1000  thrpt    5    3475.401 ±  2283.158
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS       10000  thrpt    5      56.215 ±   
11.524  ops/s

Benchmark result is saved to dist/reports/jmh/results.txt

```


Thanks,

Reza Motamedi


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message