aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Farner" <wfar...@apache.org>
Subject Re: Review Request 28731: Implemented TaskScheduler benchmarks.
Date Wed, 21 Jan 2015 23:48:59 GMT


> On Jan. 21, 2015, 10:27 p.m., Bill Farner wrote:
> > Meta question - how do you envision us using this?  When a perf issue is discovered,
should we generally push for a test case to be added here to validate the perf fix?
> 
> Maxim Khutornenko wrote:
>     I doubt we can build a reliable/automated way to catch regressions due to perf. There
is too much randomness coming from JVM and machine load for the scattered test runs to be
comparable. These bencmarks are most useful when run back to back against different builds.
The way I see it is more of a fix validation testbed than a comprehensive perf regression
framework. 
>     
>     To your question, new tests may be required if the fix validation requires cluster/data
setup significantly different from the existing tests. Otherwise, running the existing tests
before and after the fix should be sufficient.

Agreed - absolute figures are not necessary useful such that we will institute thresholds
for passage.


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28731/#review69047
-----------------------------------------------------------


On Jan. 7, 2015, 1:34 a.m., Maxim Khutornenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28731/
> -----------------------------------------------------------
> 
> (Updated Jan. 7, 2015, 1:34 a.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Added baseline benchmarks for a few static veto cases.
> 
> 
> Diffs
> -----
> 
>   build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 
>   src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49

>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeClusterState.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION

>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5

>   src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c

>   src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d

> 
> Diff: https://reviews.apache.org/r/28731/diff/
> 
> 
> Testing
> -------
> 
> Sample run on a local box:
> ```
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
> # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 10 iterations, 1 s each
> # Measurement: 100 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark
> 
> # Run progress: 0.00% complete, ETA 00:05:30
> # Fork: 1 of 1
> # Warmup Iteration   1: 278284250.000 ns/op
> # Warmup Iteration   2: 70294312.500 ns/op
> # Warmup Iteration   3: 19293379.310 ns/op
> # Warmup Iteration   4: 11945387.097 ns/op
> # Warmup Iteration   5: 10725388.350 ns/op
> # Warmup Iteration   6: 13043282.353 ns/op
> # Warmup Iteration   7: 9233083.333 ns/op
> # Warmup Iteration   8: 9521051.724 ns/op
> # Warmup Iteration   9: 10750854.369 ns/op
> # Warmup Iteration  10: 7460243.243 ns/op
> Iteration   1: 7885364.286 ns/op
> Iteration   2: 7735139.860 ns/op
> Iteration   3: 7660208.333 ns/op
> Iteration   4: 7761204.225 ns/op
> Iteration   5: 7868907.143 ns/op
> Iteration   6: 7567404.110 ns/op
> Iteration   7: 7611000.000 ns/op
> Iteration   8: 7766154.930 ns/op
> Iteration   9: 7669344.828 ns/op
> Iteration  10: 7707783.217 ns/op
> Iteration  11: 7435651.007 ns/op
> Iteration  12: 7697631.944 ns/op
> Iteration  13: 7712531.469 ns/op
> Iteration  14: 7899407.143 ns/op
> Iteration  15: 7448472.973 ns/op
> Iteration  16: 7791521.127 ns/op
> Iteration  17: 7612213.793 ns/op
> Iteration  18: 7710867.133 ns/op
> Iteration  19: 7649296.552 ns/op
> Iteration  20: 7768309.859 ns/op
> Iteration  21: 7688666.667 ns/op
> Iteration  22: 7531557.823 ns/op
> Iteration  23: 7381193.333 ns/op
> Iteration  24: 7726006.993 ns/op
> Iteration  25: 7603358.621 ns/op
> Iteration  26: 7653631.944 ns/op
> Iteration  27: 7442275.168 ns/op
> Iteration  28: 7613186.207 ns/op
> Iteration  29: 7765823.944 ns/op
> Iteration  30: 7489687.075 ns/op
> Iteration  31: 7811443.662 ns/op
> Iteration  32: 8015007.246 ns/op
> Iteration  33: 8192392.593 ns/op
> Iteration  34: 8040335.766 ns/op
> Iteration  35: 7584212.329 ns/op
> Iteration  36: 8001934.783 ns/op
> Iteration  37: 9744815.789 ns/op
> Iteration  38: 11688284.211 ns/op
> Iteration  39: 8661406.250 ns/op
> Iteration  40: 7678413.793 ns/op
> Iteration  41: 8502223.077 ns/op
> Iteration  42: 7640820.690 ns/op
> Iteration  43: 7875624.113 ns/op
> Iteration  44: 7506809.524 ns/op
> Iteration  45: 8005431.655 ns/op
> Iteration  46: 8081664.234 ns/op
> Iteration  47: 7579438.356 ns/op
> Iteration  48: 7993405.797 ns/op
> Iteration  49: 7571958.904 ns/op
> Iteration  50: 8116463.235 ns/op
> Iteration  51: 7941330.935 ns/op
> Iteration  52: 7687145.833 ns/op
> Iteration  53: 8082554.745 ns/op
> Iteration  54: 7597889.655 ns/op
> Iteration  55: 7299907.285 ns/op
> Iteration  56: 7992789.855 ns/op
> Iteration  57: 7648268.966 ns/op
> Iteration  58: 7570863.014 ns/op
> Iteration  59: 7885078.571 ns/op
> Iteration  60: 7647158.621 ns/op
> Iteration  61: 7830858.156 ns/op
> Iteration  62: 7773690.141 ns/op
> Iteration  63: 7905850.000 ns/op
> Iteration  64: 7653800.000 ns/op
> Iteration  65: 7408248.322 ns/op
> Iteration  66: 7961352.518 ns/op
> Iteration  67: 7879785.714 ns/op
> Iteration  68: 8308616.541 ns/op
> Iteration  69: 7396852.349 ns/op
> Iteration  70: 8203214.815 ns/op
> Iteration  71: 7548821.918 ns/op
> Iteration  72: 7834631.206 ns/op
> Iteration  73: 7451594.595 ns/op
> Iteration  74: 7633262.069 ns/op
> Iteration  75: 8182348.148 ns/op
> Iteration  76: 7412646.667 ns/op
> Iteration  77: 7662375.000 ns/op
> Iteration  78: 7714825.175 ns/op
> Iteration  79: 7726195.804 ns/op
> Iteration  80: 7552568.493 ns/op
> Iteration  81: 7642655.172 ns/op
> Iteration  82: 7632737.931 ns/op
> Iteration  83: 7567616.438 ns/op
> Iteration  84: 7641517.241 ns/op
> Iteration  85: 7664152.778 ns/op
> Iteration  86: 8082532.847 ns/op
> Iteration  87: 8027427.536 ns/op
> Iteration  88: 7837191.489 ns/op
> Iteration  89: 7587938.356 ns/op
> Iteration  90: 7631262.069 ns/op
> Iteration  91: 7672423.611 ns/op
> Iteration  92: 7505258.503 ns/op
> Iteration  93: 7410449.664 ns/op
> Iteration  94: 8164419.118 ns/op
> Iteration  95: 7712930.070 ns/op
> Iteration  96: 7934287.770 ns/op
> Iteration  97: 8007239.130 ns/op
> Iteration  98: 8250343.284 ns/op
> Iteration  99: 8482853.846 ns/op
> Iteration 100: 8138595.588 ns/op
> 
> 
> Result: 7831342.453 ±(99.9%) 172179.897 ns/op [Average]
>   Statistics: (min, avg, max) = (7299907.285, 7831342.453, 11688284.211), stdev = 507676.348
>   Confidence interval (99.9%): [7659162.555, 8003522.350]
> 
> 
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
> # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 10 iterations, 1 s each
> # Measurement: 100 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark
> 
> # Run progress: 33.33% complete, ETA 00:04:43
> # Fork: 1 of 1
> # Warmup Iteration   1: 130891625.000 ns/op
> # Warmup Iteration   2: 18382600.000 ns/op
> # Warmup Iteration   3: 8235328.358 ns/op
> # Warmup Iteration   4: 8724039.370 ns/op
> # Warmup Iteration   5: 5823163.158 ns/op
> # Warmup Iteration   6: 5425686.275 ns/op
> # Warmup Iteration   7: 5722766.839 ns/op
> # Warmup Iteration   8: 5716419.689 ns/op
> # Warmup Iteration   9: 6278869.318 ns/op
> # Warmup Iteration  10: 5738276.042 ns/op
> Iteration   1: 5761427.083 ns/op
> Iteration   2: 5946838.710 ns/op
> Iteration   3: 6091066.298 ns/op
> Iteration   4: 5721787.565 ns/op
> Iteration   5: 6785852.761 ns/op
> Iteration   6: 5813373.684 ns/op
> Iteration   7: 5866824.468 ns/op
> Iteration   8: 6948911.950 ns/op
> Iteration   9: 6042060.109 ns/op
> Iteration  10: 5633122.449 ns/op
> Iteration  11: 5753838.542 ns/op
> Iteration  12: 5853433.862 ns/op
> Iteration  13: 5752322.917 ns/op
> Iteration  14: 5332053.140 ns/op
> Iteration  15: 5318278.846 ns/op
> Iteration  16: 5260276.190 ns/op
> Iteration  17: 5700340.206 ns/op
> Iteration  18: 5767635.417 ns/op
> Iteration  19: 5367514.563 ns/op
> Iteration  20: 5721994.819 ns/op
> Iteration  21: 5711243.523 ns/op
> Iteration  22: 6276647.727 ns/op
> Iteration  23: 5934473.118 ns/op
> Iteration  24: 5821626.316 ns/op
> Iteration  25: 5533788.945 ns/op
> Iteration  26: 7226183.007 ns/op
> Iteration  27: 5448920.792 ns/op
> Iteration  28: 5749453.125 ns/op
> Iteration  29: 5906727.273 ns/op
> Iteration  30: 5702489.691 ns/op
> Iteration  31: 5355660.194 ns/op
> Iteration  32: 7413523.490 ns/op
> Iteration  33: 7467831.081 ns/op
> Iteration  34: 8952487.805 ns/op
> Iteration  35: 8635742.188 ns/op
> Iteration  36: 7981898.551 ns/op
> Iteration  37: 7761732.394 ns/op
> Iteration  38: 7130438.710 ns/op
> Iteration  39: 8349406.015 ns/op
> Iteration  40: 7980985.612 ns/op
> Iteration  41: 8037847.826 ns/op
> Iteration  42: 6315828.571 ns/op
> Iteration  43: 5375892.683 ns/op
> Iteration  44: 5897005.348 ns/op
> Iteration  45: 5568793.970 ns/op
> Iteration  46: 5273685.714 ns/op
> Iteration  47: 8879392.000 ns/op
> Iteration  48: 8641343.750 ns/op
> Iteration  49: 10007136.364 ns/op
> Iteration  50: 8484908.397 ns/op
> Iteration  51: 7659333.333 ns/op
> Iteration  52: 8661968.750 ns/op
> Iteration  53: 8923153.226 ns/op
> Iteration  54: 8746527.559 ns/op
> Iteration  55: 8981138.211 ns/op
> Iteration  56: 5631193.878 ns/op
> Iteration  57: 5945069.892 ns/op
> Iteration  58: 5610161.616 ns/op
> Iteration  59: 5348665.049 ns/op
> Iteration  60: 5429064.039 ns/op
> Iteration  61: 5432418.719 ns/op
> Iteration  62: 5621515.306 ns/op
> Iteration  63: 5268630.332 ns/op
> Iteration  64: 5849894.180 ns/op
> Iteration  65: 5418126.829 ns/op
> Iteration  66: 6133177.778 ns/op
> Iteration  67: 6195095.506 ns/op
> Iteration  68: 6111276.243 ns/op
> Iteration  69: 5886244.681 ns/op
> Iteration  70: 6430941.860 ns/op
> Iteration  71: 5783329.843 ns/op
> Iteration  72: 6507141.176 ns/op
> Iteration  73: 10092783.784 ns/op
> Iteration  74: 11873978.495 ns/op
> Iteration  75: 8876424.000 ns/op
> Iteration  76: 5878223.404 ns/op
> Iteration  77: 7130651.613 ns/op
> Iteration  78: 5508910.000 ns/op
> Iteration  79: 5481338.308 ns/op
> Iteration  80: 5999592.391 ns/op
> Iteration  81: 6065917.582 ns/op
> Iteration  82: 6087812.155 ns/op
> Iteration  83: 5653512.821 ns/op
> Iteration  84: 5528065.000 ns/op
> Iteration  85: 5599314.721 ns/op
> Iteration  86: 5300822.115 ns/op
> Iteration  87: 6341833.333 ns/op
> Iteration  88: 5511190.000 ns/op
> Iteration  89: 5813026.316 ns/op
> Iteration  90: 5914112.299 ns/op
> Iteration  91: 5663928.205 ns/op
> Iteration  92: 5723891.192 ns/op
> Iteration  93: 5580843.434 ns/op
> Iteration  94: 5990407.609 ns/op
> Iteration  95: 5656717.949 ns/op
> Iteration  96: 5432532.020 ns/op
> Iteration  97: 5797827.225 ns/op
> Iteration  98: 5647410.256 ns/op
> Iteration  99: 5250814.286 ns/op
> Iteration 100: 5602177.665 ns/op
> 
> 
> Result: 6418401.739 ±(99.9%) 447052.433 ns/op [Average]
>   Statistics: (min, avg, max) = (5250814.286, 6418401.739, 11873978.495), stdev = 1318144.279
>   Confidence interval (99.9%): [5971349.307, 6865454.172]
> 
> 
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
> # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 10 iterations, 1 s each
> # Measurement: 100 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.PreemptorFallbackForLargeClusterBenchmark.runBenchmark
> 
> # Run progress: 66.67% complete, ETA 00:02:19
> # Fork: 1 of 1
> # Warmup Iteration   1: 7780760000.000 ns/op
> # Warmup Iteration   2: 165927428.571 ns/op
> # Warmup Iteration   3: 94242076.923 ns/op
> # Warmup Iteration   4: 66061000.000 ns/op
> # Warmup Iteration   5: 70444500.000 ns/op
> # Warmup Iteration   6: 62350000.000 ns/op
> # Warmup Iteration   7: 63746555.556 ns/op
> # Warmup Iteration   8: 62712111.111 ns/op
> # Warmup Iteration   9: 62776388.889 ns/op
> # Warmup Iteration  10: 62455277.778 ns/op
> Iteration   1: 65696764.706 ns/op
> Iteration   2: 66255823.529 ns/op
> Iteration   3: 68231647.059 ns/op
> Iteration   4: 71606750.000 ns/op
> Iteration   5: 66780235.294 ns/op
> Iteration   6: 69914750.000 ns/op
> Iteration   7: 74681333.333 ns/op
> Iteration   8: 64133944.444 ns/op
> Iteration   9: 60831736.842 ns/op
> Iteration  10: 63261444.444 ns/op
> Iteration  11: 60814105.263 ns/op
> Iteration  12: 67882941.176 ns/op
> Iteration  13: 62512888.889 ns/op
> Iteration  14: 65551823.529 ns/op
> Iteration  15: 65574294.118 ns/op
> Iteration  16: 67928352.941 ns/op
> Iteration  17: 65510588.235 ns/op
> Iteration  18: 62637055.556 ns/op
> Iteration  19: 62900222.222 ns/op
> Iteration  20: 63715944.444 ns/op
> Iteration  21: 62284611.111 ns/op
> Iteration  22: 68174235.294 ns/op
> Iteration  23: 69029187.500 ns/op
> Iteration  24: 68584764.706 ns/op
> Iteration  25: 63332777.778 ns/op
> Iteration  26: 64316000.000 ns/op
> Iteration  27: 64219444.444 ns/op
> Iteration  28: 64219888.889 ns/op
> Iteration  29: 63113888.889 ns/op
> Iteration  30: 65802117.647 ns/op
> Iteration  31: 63661277.778 ns/op
> Iteration  32: 63322000.000 ns/op
> Iteration  33: 63770000.000 ns/op
> Iteration  34: 63920333.333 ns/op
> Iteration  35: 62403500.000 ns/op
> Iteration  36: 64137166.667 ns/op
> Iteration  37: 64725444.444 ns/op
> Iteration  38: 63145166.667 ns/op
> Iteration  39: 64183944.444 ns/op
> Iteration  40: 65873666.667 ns/op
> Iteration  41: 65648529.412 ns/op
> Iteration  42: 66487176.471 ns/op
> Iteration  43: 61702888.889 ns/op
> Iteration  44: 63267666.667 ns/op
> Iteration  45: 69614125.000 ns/op
> Iteration  46: 61574222.222 ns/op
> Iteration  47: 62678611.111 ns/op
> Iteration  48: 62792277.778 ns/op
> Iteration  49: 65355352.941 ns/op
> Iteration  50: 65206823.529 ns/op
> Iteration  51: 64246722.222 ns/op
> Iteration  52: 68074823.529 ns/op
> Iteration  53: 67314823.529 ns/op
> Iteration  54: 65511882.353 ns/op
> Iteration  55: 63607722.222 ns/op
> Iteration  56: 66646529.412 ns/op
> Iteration  57: 67675764.706 ns/op
> Iteration  58: 62502388.889 ns/op
> Iteration  59: 70476625.000 ns/op
> Iteration  60: 97138500.000 ns/op
> Iteration  61: 93276461.538 ns/op
> Iteration  62: 103508818.182 ns/op
> Iteration  63: 96725916.667 ns/op
> Iteration  64: 97539500.000 ns/op
> Iteration  65: 93591461.538 ns/op
> Iteration  66: 95847083.333 ns/op
> Iteration  67: 101147666.667 ns/op
> Iteration  68: 91144230.769 ns/op
> Iteration  69: 69291375.000 ns/op
> Iteration  70: 67021941.176 ns/op
> Iteration  71: 64101166.667 ns/op
> Iteration  72: 66573823.529 ns/op
> Iteration  73: 66116235.294 ns/op
> Iteration  74: 62875611.111 ns/op
> Iteration  75: 72765812.500 ns/op
> Iteration  76: 65663176.471 ns/op
> Iteration  77: 64187000.000 ns/op
> Iteration  78: 64425444.444 ns/op
> Iteration  79: 63835111.111 ns/op
> Iteration  80: 62942166.667 ns/op
> Iteration  81: 69006500.000 ns/op
> Iteration  82: 64798176.471 ns/op
> Iteration  83: 67823117.647 ns/op
> Iteration  84: 77090866.667 ns/op
> Iteration  85: 71548625.000 ns/op
> Iteration  86: 63718333.333 ns/op
> Iteration  87: 67034764.706 ns/op
> Iteration  88: 61701833.333 ns/op
> Iteration  89: 65456529.412 ns/op
> Iteration  90: 63462388.889 ns/op
> Iteration  91: 61797555.556 ns/op
> Iteration  92: 66772823.529 ns/op
> Iteration  93: 66240117.647 ns/op
> Iteration  94: 63701444.444 ns/op
> Iteration  95: 64047888.889 ns/op
> Iteration  96: 62162777.778 ns/op
> Iteration  97: 64846588.235 ns/op
> Iteration  98: 61865666.667 ns/op
> Iteration  99: 61965222.222 ns/op
> Iteration 100: 64089333.333 ns/op
> 
> 
> Result: 68178760.746 ±(99.9%) 3225821.640 ns/op [Average]
>   Statistics: (min, avg, max) = (60814105.263, 68178760.746, 103508818.182), stdev =
9511408.567
>   Confidence interval (99.9%): [64952939.106, 71404582.386]
> 
> 
> # Run complete. Total time: 00:07:18
> 
> Benchmark                                                                           
  Mode  Samples         Score         Error  Units
> o.a.a.b.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark    
  avgt      100   7831342.453 ±  172179.897  ns/op
> o.a.a.b.SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark  
  avgt      100   6418401.739 ±  447052.433  ns/op
> o.a.a.b.SchedulingBenchmarks.PreemptorFallbackForLargeClusterBenchmark.runBenchmark 
  avgt      100  68178760.746 ± 3225821.640  ns/op
> ```
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message