aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mehrdad Nurolahzade <mehr...@nurolahzade.com>
Subject Re: Review Request 55284: Benchmarking SLA Algorithms
Date Tue, 10 Jan 2017 17:51:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55284/#review161084
-----------------------------------------------------------




src/jmh/java/org/apache/aurora/benchmark/Tasks.java (lines 98 - 105)
<https://reviews.apache.org/r/55284/#comment232326>

    I see what you are trying to keep task status and events consistant. But if I set the
task events first and then set the status then I'll lose the events. Also setting status based
on event timestamps is rather shaky (all those events are likely to have the same timestamp).
    
    So, I would suggest moving the consistency checking logic to `build()` method and only
verifying that the last event (last element of the `events` list) matches the status.


- Mehrdad Nurolahzade


On Jan. 10, 2017, 1 a.m., Reza Motamedi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55284/
> -----------------------------------------------------------
> 
> (Updated Jan. 10, 2017, 1 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> # Adding SLA benchmarks benchmarks 
> This RB is related to [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854).

> The goal is to add JMH benchmarks for SlaCalculation and then other metric calculators.
> 
> 
> Diffs
> -----
> 
>   src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 3aabc5ef222a97d8ebde693c52030a66bca340d2

>   src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 5d8d5bd8f705770979f284d26d2e932aabe707e5

> 
> Diff: https://reviews.apache.org/r/55284/diff/
> 
> 
> Testing
> -------
> 
> the content of <./aurora/dist/reports/jmh/human.txt>
> 
> ```
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 1000)
> 
> # Run progress: 0.00% complete, ETA 00:07:00
> # Fork: 1 of 1
> # Warmup Iteration   1: 5141.403 ops/s
> Iteration   1: 5514.591 ops/s
> Iteration   2: 5508.875 ops/s
> Iteration   3: 5594.224 ops/s
> Iteration   4: 5467.605 ops/s
> Iteration   5: 5619.229 ops/s
> 
> 
> Result "measureMTTA":
>   5540.905 ±(99.9%) 244.059 ops/s [Average]
>   (min, avg, max) = (5467.605, 5540.905, 5619.229), stdev = 63.381
>   CI (99.9%): [5296.846, 5784.963] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 10000)
> 
> # Run progress: 8.33% complete, ETA 00:06:34
> # Fork: 1 of 1
> # Warmup Iteration   1: 64.993 ops/s
> Iteration   1: 61.762 ops/s
> Iteration   2: 64.468 ops/s
> Iteration   3: 65.047 ops/s
> Iteration   4: 63.199 ops/s
> Iteration   5: 64.412 ops/s
> 
> 
> Result "measureMTTA":
>   63.778 ±(99.9%) 5.053 ops/s [Average]
>   (min, avg, max) = (61.762, 63.778, 65.047), stdev = 1.312
>   CI (99.9%): [58.725, 68.830] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 100000)
> 
> # Run progress: 16.67% complete, ETA 00:06:00
> # Fork: 1 of 1
> # Warmup Iteration   1: 0.490 ops/s
> Iteration   1: 0.556 ops/s
> Iteration   2: 0.503 ops/s
> Iteration   3: 0.502 ops/s
> Iteration   4: 0.576 ops/s
> Iteration   5: 0.591 ops/s
> 
> 
> Result "measureMTTA":
>   0.546 ±(99.9%) 0.159 ops/s [Average]
>   (min, avg, max) = (0.502, 0.546, 0.591), stdev = 0.041
>   CI (99.9%): [0.387, 0.705] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA
> # Parameters: (numTasks = 200000)
> 
> # Run progress: 25.00% complete, ETA 00:05:40
> # Fork: 1 of 1
> # Warmup Iteration   1: 0.099 ops/s
> Iteration   1: 0.130 ops/s
> Iteration   2: 0.146 ops/s
> Iteration   3: 0.117 ops/s
> Iteration   4: 0.145 ops/s
> Iteration   5: 0.144 ops/s
> 
> 
> Result "measureMTTA":
>   0.137 ±(99.9%) 0.049 ops/s [Average]
>   (min, avg, max) = (0.117, 0.137, 0.146), stdev = 0.013
>   CI (99.9%): [0.088, 0.185] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 1000)
> 
> # Run progress: 33.33% complete, ETA 00:05:28
> # Fork: 1 of 1
> # Warmup Iteration   1: 4184.816 ops/s
> Iteration   1: 4274.550 ops/s
> Iteration   2: 2630.657 ops/s
> Iteration   3: 3168.746 ops/s
> Iteration   4: 4054.913 ops/s
> Iteration   5: 3653.909 ops/s
> 
> 
> Result "measureMTTR":
>   3556.555 ±(99.9%) 2569.449 ops/s [Average]
>   (min, avg, max) = (2630.657, 3556.555, 4274.550), stdev = 667.278
>   CI (99.9%): [987.106, 6126.004] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 10000)
> 
> # Run progress: 41.67% complete, ETA 00:04:40
> # Fork: 1 of 1
> # Warmup Iteration   1: 39.150 ops/s
> Iteration   1: 56.293 ops/s
> Iteration   2: 59.521 ops/s
> Iteration   3: 46.747 ops/s
> Iteration   4: 55.249 ops/s
> Iteration   5: 52.481 ops/s
> 
> 
> Result "measureMTTR":
>   54.058 ±(99.9%) 18.486 ops/s [Average]
>   (min, avg, max) = (46.747, 54.058, 59.521), stdev = 4.801
>   CI (99.9%): [35.572, 72.544] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 100000)
> 
> # Run progress: 50.00% complete, ETA 00:03:57
> # Fork: 1 of 1
> # Warmup Iteration   1: 0.448 ops/s
> Iteration   1: 0.482 ops/s
> Iteration   2: 0.475 ops/s
> Iteration   3: 0.353 ops/s
> Iteration   4: 0.478 ops/s
> Iteration   5: 0.513 ops/s
> 
> 
> Result "measureMTTR":
>   0.460 ±(99.9%) 0.238 ops/s [Average]
>   (min, avg, max) = (0.353, 0.460, 0.513), stdev = 0.062
>   CI (99.9%): [0.223, 0.698] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR
> # Parameters: (numTasks = 200000)
> 
> # Run progress: 58.33% complete, ETA 00:03:23
> # Fork: 1 of 1
> # Warmup Iteration   1: 0.085 ops/s
> Iteration   1: 0.125 ops/s
> Iteration   2: 0.133 ops/s
> Iteration   3: 0.131 ops/s
> Iteration   4: 0.129 ops/s
> Iteration   5: 0.117 ops/s
> 
> 
> Result "measureMTTR":
>   0.127 ±(99.9%) 0.025 ops/s [Average]
>   (min, avg, max) = (0.117, 0.127, 0.133), stdev = 0.006
>   CI (99.9%): [0.102, 0.152] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 1000)
> 
> # Run progress: 66.67% complete, ETA 00:02:50
> # Fork: 1 of 1
> # Warmup Iteration   1: 3914.893 ops/s
> Iteration   1: 4447.483 ops/s
> Iteration   2: 1089.132 ops/s
> Iteration   3: 1799.594 ops/s
> Iteration   4: 4622.600 ops/s
> Iteration   5: 4259.013 ops/s
> 
> 
> Result "measureMTTS":
>   3243.565 ±(99.9%) 6417.109 ops/s [Average]
>   (min, avg, max) = (1089.132, 3243.565, 4622.600), stdev = 1666.503
>   CI (99.9%): [? 0, 9660.673] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 10000)
> 
> # Run progress: 75.00% complete, ETA 00:02:05
> # Fork: 1 of 1
> # Warmup Iteration   1: 55.750 ops/s
> Iteration   1: 56.535 ops/s
> Iteration   2: 57.740 ops/s
> Iteration   3: 59.033 ops/s
> Iteration   4: 55.974 ops/s
> Iteration   5: 49.698 ops/s
> 
> 
> Result "measureMTTS":
>   55.796 ±(99.9%) 13.886 ops/s [Average]
>   (min, avg, max) = (49.698, 55.796, 59.033), stdev = 3.606
>   CI (99.9%): [41.910, 69.682] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 100000)
> 
> # Run progress: 83.33% complete, ETA 00:01:22
> # Fork: 1 of 1
> # Warmup Iteration   1: 0.484 ops/s
> Iteration   1: 0.523 ops/s
> Iteration   2: 0.544 ops/s
> Iteration   3: 0.550 ops/s
> Iteration   4: 0.518 ops/s
> Iteration   5: 0.506 ops/s
> 
> 
> Result "measureMTTS":
>   0.528 ±(99.9%) 0.071 ops/s [Average]
>   (min, avg, max) = (0.506, 0.528, 0.550), stdev = 0.018
>   CI (99.9%): [0.458, 0.599] (assumes normal distribution)
> 
> 
> # JMH 1.15 (released 102 days ago)
> # VM version: JDK 1.8.0_111, VM 25.111-b14
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java
> # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 1 iterations, 10 s each
> # Measurement: 5 iterations, 5 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS
> # Parameters: (numTasks = 200000)
> 
> # Run progress: 91.67% complete, ETA 00:00:41
> # Fork: 1 of 1
> # Warmup Iteration   1: 0.099 ops/s
> Iteration   1: 0.131 ops/s
> Iteration   2: 0.116 ops/s
> Iteration   3: 0.114 ops/s
> Iteration   4: 0.107 ops/s
> Iteration   5: 0.105 ops/s
> 
> 
> Result "measureMTTS":
>   0.115 ±(99.9%) 0.039 ops/s [Average]
>   (min, avg, max) = (0.105, 0.115, 0.131), stdev = 0.010
>   CI (99.9%): [0.076, 0.154] (assumes normal distribution)
> 
> 
> # Run complete. Total time: 00:08:45
> 
> Benchmark                                        (numTasks)   Mode  Cnt     Score   
  Error  Units
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA        1000  thrpt    5  5540.905 ±
 244.059  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA       10000  thrpt    5    63.778 ±
   5.053  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA      100000  thrpt    5     0.546 ±
   0.159  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA      200000  thrpt    5     0.137 ±
   0.049  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR        1000  thrpt    5  3556.555 ±
2569.449  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR       10000  thrpt    5    54.058 ±
  18.486  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR      100000  thrpt    5     0.460 ±
   0.238  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR      200000  thrpt    5     0.127 ±
   0.025  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS        1000  thrpt    5  3243.565 ±
6417.109  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS       10000  thrpt    5    55.796 ±
  13.886  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS      100000  thrpt    5     0.528 ±
   0.071  ops/s
> SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS      200000  thrpt    5     0.115 ±
   0.039  ops/s
> 
> Benchmark result is saved to dist/reports/jmh/results.txt
> 
> 
> ```
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message