aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reza Motamedi <reza.motam...@gmail.com>
Subject Re: Review Request 55284: Benchmarking
Date Tue, 10 Jan 2017 09:00:09 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55284/
-----------------------------------------------------------

(Updated Jan. 10, 2017, 9 a.m.)


Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh Kumar Shanmugham.


Changes
-------

- partially refactored Task Builder to accept a list of TaskEvents.
- updated the number of tasks to show a mathc a more reasonable load.

```
Benchmark                                        (numTasks)   Mode  Cnt     Score      Error
 Units
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA        1000  thrpt    5  5540.905 ±  244.059
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA       10000  thrpt    5    63.778 ±    5.053
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA      100000  thrpt    5     0.546 ±    0.159
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA      200000  thrpt    5     0.137 ±    0.049
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR        1000  thrpt    5  3556.555 ± 2569.449
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR       10000  thrpt    5    54.058 ±   18.486
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR      100000  thrpt    5     0.460 ±    0.238
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR      200000  thrpt    5     0.127 ±    0.025
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS        1000  thrpt    5  3243.565 ± 6417.109
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS       10000  thrpt    5    55.796 ±   13.886
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS      100000  thrpt    5     0.528 ±    0.071
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS      200000  thrpt    5     0.115 ±    0.039
 ops/s

```


Repository: aurora


Description (updated)
-------

# Adding SLA benchmarks benchmarks 
This RB is related to [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854). 
The goal is to add JMH benchmarks for SlaCalculation and then other metric calculators.


Diffs (updated)
-----

  src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java PRE-CREATION 
  src/jmh/java/org/apache/aurora/benchmark/Tasks.java 3aabc5ef222a97d8ebde693c52030a66bca340d2

  src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 5d8d5bd8f705770979f284d26d2e932aabe707e5


Diff: https://reviews.apache.org/r/55284/diff/


Testing (updated)
-------

the content of <./aurora/dist/reports/jmh/human.txt>

```
# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 1000)

# Run progress: 0.00% complete, ETA 00:07:00
# Fork: 1 of 1
# Warmup Iteration   1: 5141.403 ops/s
Iteration   1: 5514.591 ops/s
Iteration   2: 5508.875 ops/s
Iteration   3: 5594.224 ops/s
Iteration   4: 5467.605 ops/s
Iteration   5: 5619.229 ops/s


Result "measureMTTA":
  5540.905 ±(99.9%) 244.059 ops/s [Average]
  (min, avg, max) = (5467.605, 5540.905, 5619.229), stdev = 63.381
  CI (99.9%): [5296.846, 5784.963] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 10000)

# Run progress: 8.33% complete, ETA 00:06:34
# Fork: 1 of 1
# Warmup Iteration   1: 64.993 ops/s
Iteration   1: 61.762 ops/s
Iteration   2: 64.468 ops/s
Iteration   3: 65.047 ops/s
Iteration   4: 63.199 ops/s
Iteration   5: 64.412 ops/s


Result "measureMTTA":
  63.778 ±(99.9%) 5.053 ops/s [Average]
  (min, avg, max) = (61.762, 63.778, 65.047), stdev = 1.312
  CI (99.9%): [58.725, 68.830] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 100000)

# Run progress: 16.67% complete, ETA 00:06:00
# Fork: 1 of 1
# Warmup Iteration   1: 0.490 ops/s
Iteration   1: 0.556 ops/s
Iteration   2: 0.503 ops/s
Iteration   3: 0.502 ops/s
Iteration   4: 0.576 ops/s
Iteration   5: 0.591 ops/s


Result "measureMTTA":
  0.546 ±(99.9%) 0.159 ops/s [Average]
  (min, avg, max) = (0.502, 0.546, 0.591), stdev = 0.041
  CI (99.9%): [0.387, 0.705] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
# Parameters: (numTasks = 200000)

# Run progress: 25.00% complete, ETA 00:05:40
# Fork: 1 of 1
# Warmup Iteration   1: 0.099 ops/s
Iteration   1: 0.130 ops/s
Iteration   2: 0.146 ops/s
Iteration   3: 0.117 ops/s
Iteration   4: 0.145 ops/s
Iteration   5: 0.144 ops/s


Result "measureMTTA":
  0.137 ±(99.9%) 0.049 ops/s [Average]
  (min, avg, max) = (0.117, 0.137, 0.146), stdev = 0.013
  CI (99.9%): [0.088, 0.185] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 1000)

# Run progress: 33.33% complete, ETA 00:05:28
# Fork: 1 of 1
# Warmup Iteration   1: 4184.816 ops/s
Iteration   1: 4274.550 ops/s
Iteration   2: 2630.657 ops/s
Iteration   3: 3168.746 ops/s
Iteration   4: 4054.913 ops/s
Iteration   5: 3653.909 ops/s


Result "measureMTTR":
  3556.555 ±(99.9%) 2569.449 ops/s [Average]
  (min, avg, max) = (2630.657, 3556.555, 4274.550), stdev = 667.278
  CI (99.9%): [987.106, 6126.004] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 10000)

# Run progress: 41.67% complete, ETA 00:04:40
# Fork: 1 of 1
# Warmup Iteration   1: 39.150 ops/s
Iteration   1: 56.293 ops/s
Iteration   2: 59.521 ops/s
Iteration   3: 46.747 ops/s
Iteration   4: 55.249 ops/s
Iteration   5: 52.481 ops/s


Result "measureMTTR":
  54.058 ±(99.9%) 18.486 ops/s [Average]
  (min, avg, max) = (46.747, 54.058, 59.521), stdev = 4.801
  CI (99.9%): [35.572, 72.544] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 100000)

# Run progress: 50.00% complete, ETA 00:03:57
# Fork: 1 of 1
# Warmup Iteration   1: 0.448 ops/s
Iteration   1: 0.482 ops/s
Iteration   2: 0.475 ops/s
Iteration   3: 0.353 ops/s
Iteration   4: 0.478 ops/s
Iteration   5: 0.513 ops/s


Result "measureMTTR":
  0.460 ±(99.9%) 0.238 ops/s [Average]
  (min, avg, max) = (0.353, 0.460, 0.513), stdev = 0.062
  CI (99.9%): [0.223, 0.698] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
# Parameters: (numTasks = 200000)

# Run progress: 58.33% complete, ETA 00:03:23
# Fork: 1 of 1
# Warmup Iteration   1: 0.085 ops/s
Iteration   1: 0.125 ops/s
Iteration   2: 0.133 ops/s
Iteration   3: 0.131 ops/s
Iteration   4: 0.129 ops/s
Iteration   5: 0.117 ops/s


Result "measureMTTR":
  0.127 ±(99.9%) 0.025 ops/s [Average]
  (min, avg, max) = (0.117, 0.127, 0.133), stdev = 0.006
  CI (99.9%): [0.102, 0.152] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 1000)

# Run progress: 66.67% complete, ETA 00:02:50
# Fork: 1 of 1
# Warmup Iteration   1: 3914.893 ops/s
Iteration   1: 4447.483 ops/s
Iteration   2: 1089.132 ops/s
Iteration   3: 1799.594 ops/s
Iteration   4: 4622.600 ops/s
Iteration   5: 4259.013 ops/s


Result "measureMTTS":
  3243.565 ±(99.9%) 6417.109 ops/s [Average]
  (min, avg, max) = (1089.132, 3243.565, 4622.600), stdev = 1666.503
  CI (99.9%): [? 0, 9660.673] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 10000)

# Run progress: 75.00% complete, ETA 00:02:05
# Fork: 1 of 1
# Warmup Iteration   1: 55.750 ops/s
Iteration   1: 56.535 ops/s
Iteration   2: 57.740 ops/s
Iteration   3: 59.033 ops/s
Iteration   4: 55.974 ops/s
Iteration   5: 49.698 ops/s


Result "measureMTTS":
  55.796 ±(99.9%) 13.886 ops/s [Average]
  (min, avg, max) = (49.698, 55.796, 59.033), stdev = 3.606
  CI (99.9%): [41.910, 69.682] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 100000)

# Run progress: 83.33% complete, ETA 00:01:22
# Fork: 1 of 1
# Warmup Iteration   1: 0.484 ops/s
Iteration   1: 0.523 ops/s
Iteration   2: 0.544 ops/s
Iteration   3: 0.550 ops/s
Iteration   4: 0.518 ops/s
Iteration   5: 0.506 ops/s


Result "measureMTTS":
  0.528 ±(99.9%) 0.071 ops/s [Average]
  (min, avg, max) = (0.506, 0.528, 0.550), stdev = 0.018
  CI (99.9%): [0.458, 0.599] (assumes normal distribution)


# JMH 1.15 (released 102 days ago)
# VM version: JDK 1.8.0_111, VM 25.111-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
# VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Warmup: 1 iterations, 10 s each
# Measurement: 5 iterations, 5 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
# Parameters: (numTasks = 200000)

# Run progress: 91.67% complete, ETA 00:00:41
# Fork: 1 of 1
# Warmup Iteration   1: 0.099 ops/s
Iteration   1: 0.131 ops/s
Iteration   2: 0.116 ops/s
Iteration   3: 0.114 ops/s
Iteration   4: 0.107 ops/s
Iteration   5: 0.105 ops/s


Result "measureMTTS":
  0.115 ±(99.9%) 0.039 ops/s [Average]
  (min, avg, max) = (0.105, 0.115, 0.131), stdev = 0.010
  CI (99.9%): [0.076, 0.154] (assumes normal distribution)


# Run complete. Total time: 00:08:45

Benchmark                                        (numTasks)   Mode  Cnt     Score      Error
 Units
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA        1000  thrpt    5  5540.905 ±  244.059
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA       10000  thrpt    5    63.778 ±    5.053
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA      100000  thrpt    5     0.546 ±    0.159
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA      200000  thrpt    5     0.137 ±    0.049
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR        1000  thrpt    5  3556.555 ± 2569.449
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR       10000  thrpt    5    54.058 ±   18.486
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR      100000  thrpt    5     0.460 ±    0.238
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR      200000  thrpt    5     0.127 ±    0.025
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS        1000  thrpt    5  3243.565 ± 6417.109
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS       10000  thrpt    5    55.796 ±   13.886
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS      100000  thrpt    5     0.528 ±    0.071
 ops/s
SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS      200000  thrpt    5     0.115 ±    0.039
 ops/s

Benchmark result is saved to dist/reports/jmh/results.txt


```


Thanks,

Reza Motamedi


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message