Mailing-List: contact reviews-help@aurora.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: reviews@aurora.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============3353749908454892176=="
MIME-Version: 1.0
Subject: Re: Review Request 28731: Implemented TaskScheduler benchmarks.
From: "Bill Farner" <wfarner@apache.org>
To: "Bill Farner" <wfarner@apache.org>, "Kevin Sweeney" <kevints@apache.org>
Cc: "Aurora" <reviews@aurora.incubator.apache.org>,
 "Maxim Khutornenko" <maxim@apache.org>
Date: Wed, 21 Jan 2015 18:19:40 -0000
Message-ID: <20150121181940.9402.99584@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Bill Farner" <noreply@reviews.apache.org>
References: <20150107013414.26560.28276@reviews.apache.org>
In-Reply-To: <20150107013414.26560.28276@reviews.apache.org>
Reply-To: "Bill Farner" <wfarner@apache.org>

--===============3353749908454892176==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28731/#review68950
-----------------------------------------------------------


Kevin - any chance you're available to take a pass at this soon?  My attempts to free up a chunk of time for a thoughtful review have so far failed, and this is blocking other work of Maxim's.

- Bill Farner


On Jan. 7, 2015, 1:34 a.m., Maxim Khutornenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28731/
> -----------------------------------------------------------
> 
> (Updated Jan. 7, 2015, 1:34 a.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Added baseline benchmarks for a few static veto cases.
> 
> 
> Diffs
> -----
> 
>   build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 
>   src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeClusterState.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 
>   src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c 
>   src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d 
> 
> Diff: https://reviews.apache.org/r/28731/diff/
> 
> 
> Testing
> -------
> 
> Sample run on a local box:
> ```
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
> # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 10 iterations, 1 s each
> # Measurement: 100 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark
> 
> # Run progress: 0.00% complete, ETA 00:05:30
> # Fork: 1 of 1
> # Warmup Iteration   1: 278284250.000 ns/op
> # Warmup Iteration   2: 70294312.500 ns/op
> # Warmup Iteration   3: 19293379.310 ns/op
> # Warmup Iteration   4: 11945387.097 ns/op
> # Warmup Iteration   5: 10725388.350 ns/op
> # Warmup Iteration   6: 13043282.353 ns/op
> # Warmup Iteration   7: 9233083.333 ns/op
> # Warmup Iteration   8: 9521051.724 ns/op
> # Warmup Iteration   9: 10750854.369 ns/op
> # Warmup Iteration  10: 7460243.243 ns/op
> Iteration   1: 7885364.286 ns/op
> Iteration   2: 7735139.860 ns/op
> Iteration   3: 7660208.333 ns/op
> Iteration   4: 7761204.225 ns/op
> Iteration   5: 7868907.143 ns/op
> Iteration   6: 7567404.110 ns/op
> Iteration   7: 7611000.000 ns/op
> Iteration   8: 7766154.930 ns/op
> Iteration   9: 7669344.828 ns/op
> Iteration  10: 7707783.217 ns/op
> Iteration  11: 7435651.007 ns/op
> Iteration  12: 7697631.944 ns/op
> Iteration  13: 7712531.469 ns/op
> Iteration  14: 7899407.143 ns/op
> Iteration  15: 7448472.973 ns/op
> Iteration  16: 7791521.127 ns/op
> Iteration  17: 7612213.793 ns/op
> Iteration  18: 7710867.133 ns/op
> Iteration  19: 7649296.552 ns/op
> Iteration  20: 7768309.859 ns/op
> Iteration  21: 7688666.667 ns/op
> Iteration  22: 7531557.823 ns/op
> Iteration  23: 7381193.333 ns/op
> Iteration  24: 7726006.993 ns/op
> Iteration  25: 7603358.621 ns/op
> Iteration  26: 7653631.944 ns/op
> Iteration  27: 7442275.168 ns/op
> Iteration  28: 7613186.207 ns/op
> Iteration  29: 7765823.944 ns/op
> Iteration  30: 7489687.075 ns/op
> Iteration  31: 7811443.662 ns/op
> Iteration  32: 8015007.246 ns/op
> Iteration  33: 8192392.593 ns/op
> Iteration  34: 8040335.766 ns/op
> Iteration  35: 7584212.329 ns/op
> Iteration  36: 8001934.783 ns/op
> Iteration  37: 9744815.789 ns/op
> Iteration  38: 11688284.211 ns/op
> Iteration  39: 8661406.250 ns/op
> Iteration  40: 7678413.793 ns/op
> Iteration  41: 8502223.077 ns/op
> Iteration  42: 7640820.690 ns/op
> Iteration  43: 7875624.113 ns/op
> Iteration  44: 7506809.524 ns/op
> Iteration  45: 8005431.655 ns/op
> Iteration  46: 8081664.234 ns/op
> Iteration  47: 7579438.356 ns/op
> Iteration  48: 7993405.797 ns/op
> Iteration  49: 7571958.904 ns/op
> Iteration  50: 8116463.235 ns/op
> Iteration  51: 7941330.935 ns/op
> Iteration  52: 7687145.833 ns/op
> Iteration  53: 8082554.745 ns/op
> Iteration  54: 7597889.655 ns/op
> Iteration  55: 7299907.285 ns/op
> Iteration  56: 7992789.855 ns/op
> Iteration  57: 7648268.966 ns/op
> Iteration  58: 7570863.014 ns/op
> Iteration  59: 7885078.571 ns/op
> Iteration  60: 7647158.621 ns/op
> Iteration  61: 7830858.156 ns/op
> Iteration  62: 7773690.141 ns/op
> Iteration  63: 7905850.000 ns/op
> Iteration  64: 7653800.000 ns/op
> Iteration  65: 7408248.322 ns/op
> Iteration  66: 7961352.518 ns/op
> Iteration  67: 7879785.714 ns/op
> Iteration  68: 8308616.541 ns/op
> Iteration  69: 7396852.349 ns/op
> Iteration  70: 8203214.815 ns/op
> Iteration  71: 7548821.918 ns/op
> Iteration  72: 7834631.206 ns/op
> Iteration  73: 7451594.595 ns/op
> Iteration  74: 7633262.069 ns/op
> Iteration  75: 8182348.148 ns/op
> Iteration  76: 7412646.667 ns/op
> Iteration  77: 7662375.000 ns/op
> Iteration  78: 7714825.175 ns/op
> Iteration  79: 7726195.804 ns/op
> Iteration  80: 7552568.493 ns/op
> Iteration  81: 7642655.172 ns/op
> Iteration  82: 7632737.931 ns/op
> Iteration  83: 7567616.438 ns/op
> Iteration  84: 7641517.241 ns/op
> Iteration  85: 7664152.778 ns/op
> Iteration  86: 8082532.847 ns/op
> Iteration  87: 8027427.536 ns/op
> Iteration  88: 7837191.489 ns/op
> Iteration  89: 7587938.356 ns/op
> Iteration  90: 7631262.069 ns/op
> Iteration  91: 7672423.611 ns/op
> Iteration  92: 7505258.503 ns/op
> Iteration  93: 7410449.664 ns/op
> Iteration  94: 8164419.118 ns/op
> Iteration  95: 7712930.070 ns/op
> Iteration  96: 7934287.770 ns/op
> Iteration  97: 8007239.130 ns/op
> Iteration  98: 8250343.284 ns/op
> Iteration  99: 8482853.846 ns/op
> Iteration 100: 8138595.588 ns/op
> 
> 
> Result: 7831342.453 Â±(99.9%) 172179.897 ns/op [Average]
>   Statistics: (min, avg, max) = (7299907.285, 7831342.453, 11688284.211), stdev = 507676.348
>   Confidence interval (99.9%): [7659162.555, 8003522.350]
> 
> 
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
> # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 10 iterations, 1 s each
> # Measurement: 100 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark
> 
> # Run progress: 33.33% complete, ETA 00:04:43
> # Fork: 1 of 1
> # Warmup Iteration   1: 130891625.000 ns/op
> # Warmup Iteration   2: 18382600.000 ns/op
> # Warmup Iteration   3: 8235328.358 ns/op
> # Warmup Iteration   4: 8724039.370 ns/op
> # Warmup Iteration   5: 5823163.158 ns/op
> # Warmup Iteration   6: 5425686.275 ns/op
> # Warmup Iteration   7: 5722766.839 ns/op
> # Warmup Iteration   8: 5716419.689 ns/op
> # Warmup Iteration   9: 6278869.318 ns/op
> # Warmup Iteration  10: 5738276.042 ns/op
> Iteration   1: 5761427.083 ns/op
> Iteration   2: 5946838.710 ns/op
> Iteration   3: 6091066.298 ns/op
> Iteration   4: 5721787.565 ns/op
> Iteration   5: 6785852.761 ns/op
> Iteration   6: 5813373.684 ns/op
> Iteration   7: 5866824.468 ns/op
> Iteration   8: 6948911.950 ns/op
> Iteration   9: 6042060.109 ns/op
> Iteration  10: 5633122.449 ns/op
> Iteration  11: 5753838.542 ns/op
> Iteration  12: 5853433.862 ns/op
> Iteration  13: 5752322.917 ns/op
> Iteration  14: 5332053.140 ns/op
> Iteration  15: 5318278.846 ns/op
> Iteration  16: 5260276.190 ns/op
> Iteration  17: 5700340.206 ns/op
> Iteration  18: 5767635.417 ns/op
> Iteration  19: 5367514.563 ns/op
> Iteration  20: 5721994.819 ns/op
> Iteration  21: 5711243.523 ns/op
> Iteration  22: 6276647.727 ns/op
> Iteration  23: 5934473.118 ns/op
> Iteration  24: 5821626.316 ns/op
> Iteration  25: 5533788.945 ns/op
> Iteration  26: 7226183.007 ns/op
> Iteration  27: 5448920.792 ns/op
> Iteration  28: 5749453.125 ns/op
> Iteration  29: 5906727.273 ns/op
> Iteration  30: 5702489.691 ns/op
> Iteration  31: 5355660.194 ns/op
> Iteration  32: 7413523.490 ns/op
> Iteration  33: 7467831.081 ns/op
> Iteration  34: 8952487.805 ns/op
> Iteration  35: 8635742.188 ns/op
> Iteration  36: 7981898.551 ns/op
> Iteration  37: 7761732.394 ns/op
> Iteration  38: 7130438.710 ns/op
> Iteration  39: 8349406.015 ns/op
> Iteration  40: 7980985.612 ns/op
> Iteration  41: 8037847.826 ns/op
> Iteration  42: 6315828.571 ns/op
> Iteration  43: 5375892.683 ns/op
> Iteration  44: 5897005.348 ns/op
> Iteration  45: 5568793.970 ns/op
> Iteration  46: 5273685.714 ns/op
> Iteration  47: 8879392.000 ns/op
> Iteration  48: 8641343.750 ns/op
> Iteration  49: 10007136.364 ns/op
> Iteration  50: 8484908.397 ns/op
> Iteration  51: 7659333.333 ns/op
> Iteration  52: 8661968.750 ns/op
> Iteration  53: 8923153.226 ns/op
> Iteration  54: 8746527.559 ns/op
> Iteration  55: 8981138.211 ns/op
> Iteration  56: 5631193.878 ns/op
> Iteration  57: 5945069.892 ns/op
> Iteration  58: 5610161.616 ns/op
> Iteration  59: 5348665.049 ns/op
> Iteration  60: 5429064.039 ns/op
> Iteration  61: 5432418.719 ns/op
> Iteration  62: 5621515.306 ns/op
> Iteration  63: 5268630.332 ns/op
> Iteration  64: 5849894.180 ns/op
> Iteration  65: 5418126.829 ns/op
> Iteration  66: 6133177.778 ns/op
> Iteration  67: 6195095.506 ns/op
> Iteration  68: 6111276.243 ns/op
> Iteration  69: 5886244.681 ns/op
> Iteration  70: 6430941.860 ns/op
> Iteration  71: 5783329.843 ns/op
> Iteration  72: 6507141.176 ns/op
> Iteration  73: 10092783.784 ns/op
> Iteration  74: 11873978.495 ns/op
> Iteration  75: 8876424.000 ns/op
> Iteration  76: 5878223.404 ns/op
> Iteration  77: 7130651.613 ns/op
> Iteration  78: 5508910.000 ns/op
> Iteration  79: 5481338.308 ns/op
> Iteration  80: 5999592.391 ns/op
> Iteration  81: 6065917.582 ns/op
> Iteration  82: 6087812.155 ns/op
> Iteration  83: 5653512.821 ns/op
> Iteration  84: 5528065.000 ns/op
> Iteration  85: 5599314.721 ns/op
> Iteration  86: 5300822.115 ns/op
> Iteration  87: 6341833.333 ns/op
> Iteration  88: 5511190.000 ns/op
> Iteration  89: 5813026.316 ns/op
> Iteration  90: 5914112.299 ns/op
> Iteration  91: 5663928.205 ns/op
> Iteration  92: 5723891.192 ns/op
> Iteration  93: 5580843.434 ns/op
> Iteration  94: 5990407.609 ns/op
> Iteration  95: 5656717.949 ns/op
> Iteration  96: 5432532.020 ns/op
> Iteration  97: 5797827.225 ns/op
> Iteration  98: 5647410.256 ns/op
> Iteration  99: 5250814.286 ns/op
> Iteration 100: 5602177.665 ns/op
> 
> 
> Result: 6418401.739 Â±(99.9%) 447052.433 ns/op [Average]
>   Statistics: (min, avg, max) = (5250814.286, 6418401.739, 11873978.495), stdev = 1318144.279
>   Confidence interval (99.9%): [5971349.307, 6865454.172]
> 
> 
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
> # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
> # Warmup: 10 iterations, 1 s each
> # Measurement: 100 iterations, 1 s each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Average time, time/op
> # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.PreemptorFallbackForLargeClusterBenchmark.runBenchmark
> 
> # Run progress: 66.67% complete, ETA 00:02:19
> # Fork: 1 of 1
> # Warmup Iteration   1: 7780760000.000 ns/op
> # Warmup Iteration   2: 165927428.571 ns/op
> # Warmup Iteration   3: 94242076.923 ns/op
> # Warmup Iteration   4: 66061000.000 ns/op
> # Warmup Iteration   5: 70444500.000 ns/op
> # Warmup Iteration   6: 62350000.000 ns/op
> # Warmup Iteration   7: 63746555.556 ns/op
> # Warmup Iteration   8: 62712111.111 ns/op
> # Warmup Iteration   9: 62776388.889 ns/op
> # Warmup Iteration  10: 62455277.778 ns/op
> Iteration   1: 65696764.706 ns/op
> Iteration   2: 66255823.529 ns/op
> Iteration   3: 68231647.059 ns/op
> Iteration   4: 71606750.000 ns/op
> Iteration   5: 66780235.294 ns/op
> Iteration   6: 69914750.000 ns/op
> Iteration   7: 74681333.333 ns/op
> Iteration   8: 64133944.444 ns/op
> Iteration   9: 60831736.842 ns/op
> Iteration  10: 63261444.444 ns/op
> Iteration  11: 60814105.263 ns/op
> Iteration  12: 67882941.176 ns/op
> Iteration  13: 62512888.889 ns/op
> Iteration  14: 65551823.529 ns/op
> Iteration  15: 65574294.118 ns/op
> Iteration  16: 67928352.941 ns/op
> Iteration  17: 65510588.235 ns/op
> Iteration  18: 62637055.556 ns/op
> Iteration  19: 62900222.222 ns/op
> Iteration  20: 63715944.444 ns/op
> Iteration  21: 62284611.111 ns/op
> Iteration  22: 68174235.294 ns/op
> Iteration  23: 69029187.500 ns/op
> Iteration  24: 68584764.706 ns/op
> Iteration  25: 63332777.778 ns/op
> Iteration  26: 64316000.000 ns/op
> Iteration  27: 64219444.444 ns/op
> Iteration  28: 64219888.889 ns/op
> Iteration  29: 63113888.889 ns/op
> Iteration  30: 65802117.647 ns/op
> Iteration  31: 63661277.778 ns/op
> Iteration  32: 63322000.000 ns/op
> Iteration  33: 63770000.000 ns/op
> Iteration  34: 63920333.333 ns/op
> Iteration  35: 62403500.000 ns/op
> Iteration  36: 64137166.667 ns/op
> Iteration  37: 64725444.444 ns/op
> Iteration  38: 63145166.667 ns/op
> Iteration  39: 64183944.444 ns/op
> Iteration  40: 65873666.667 ns/op
> Iteration  41: 65648529.412 ns/op
> Iteration  42: 66487176.471 ns/op
> Iteration  43: 61702888.889 ns/op
> Iteration  44: 63267666.667 ns/op
> Iteration  45: 69614125.000 ns/op
> Iteration  46: 61574222.222 ns/op
> Iteration  47: 62678611.111 ns/op
> Iteration  48: 62792277.778 ns/op
> Iteration  49: 65355352.941 ns/op
> Iteration  50: 65206823.529 ns/op
> Iteration  51: 64246722.222 ns/op
> Iteration  52: 68074823.529 ns/op
> Iteration  53: 67314823.529 ns/op
> Iteration  54: 65511882.353 ns/op
> Iteration  55: 63607722.222 ns/op
> Iteration  56: 66646529.412 ns/op
> Iteration  57: 67675764.706 ns/op
> Iteration  58: 62502388.889 ns/op
> Iteration  59: 70476625.000 ns/op
> Iteration  60: 97138500.000 ns/op
> Iteration  61: 93276461.538 ns/op
> Iteration  62: 103508818.182 ns/op
> Iteration  63: 96725916.667 ns/op
> Iteration  64: 97539500.000 ns/op
> Iteration  65: 93591461.538 ns/op
> Iteration  66: 95847083.333 ns/op
> Iteration  67: 101147666.667 ns/op
> Iteration  68: 91144230.769 ns/op
> Iteration  69: 69291375.000 ns/op
> Iteration  70: 67021941.176 ns/op
> Iteration  71: 64101166.667 ns/op
> Iteration  72: 66573823.529 ns/op
> Iteration  73: 66116235.294 ns/op
> Iteration  74: 62875611.111 ns/op
> Iteration  75: 72765812.500 ns/op
> Iteration  76: 65663176.471 ns/op
> Iteration  77: 64187000.000 ns/op
> Iteration  78: 64425444.444 ns/op
> Iteration  79: 63835111.111 ns/op
> Iteration  80: 62942166.667 ns/op
> Iteration  81: 69006500.000 ns/op
> Iteration  82: 64798176.471 ns/op
> Iteration  83: 67823117.647 ns/op
> Iteration  84: 77090866.667 ns/op
> Iteration  85: 71548625.000 ns/op
> Iteration  86: 63718333.333 ns/op
> Iteration  87: 67034764.706 ns/op
> Iteration  88: 61701833.333 ns/op
> Iteration  89: 65456529.412 ns/op
> Iteration  90: 63462388.889 ns/op
> Iteration  91: 61797555.556 ns/op
> Iteration  92: 66772823.529 ns/op
> Iteration  93: 66240117.647 ns/op
> Iteration  94: 63701444.444 ns/op
> Iteration  95: 64047888.889 ns/op
> Iteration  96: 62162777.778 ns/op
> Iteration  97: 64846588.235 ns/op
> Iteration  98: 61865666.667 ns/op
> Iteration  99: 61965222.222 ns/op
> Iteration 100: 64089333.333 ns/op
> 
> 
> Result: 68178760.746 Â±(99.9%) 3225821.640 ns/op [Average]
>   Statistics: (min, avg, max) = (60814105.263, 68178760.746, 103508818.182), stdev = 9511408.567
>   Confidence interval (99.9%): [64952939.106, 71404582.386]
> 
> 
> # Run complete. Total time: 00:07:18
> 
> Benchmark                                                                              Mode  Samples         Score         Error  Units
> o.a.a.b.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark       avgt      100   7831342.453 Â±  172179.897  ns/op
> o.a.a.b.SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark     avgt      100   6418401.739 Â±  447052.433  ns/op
> o.a.a.b.SchedulingBenchmarks.PreemptorFallbackForLargeClusterBenchmark.runBenchmark    avgt      100  68178760.746 Â± 3225821.640  ns/op
> ```
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>


--===============3353749908454892176==--