geode-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinmei Liao (Jira)" <j...@apache.org>
Subject [jira] [Created] (GEODE-7231) CI: BENCHMARK FAILED: org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark average latency is 5% worse than baseline.
Date Mon, 23 Sep 2019 16:53:00 GMT
Jinmei Liao created GEODE-7231:
----------------------------------

             Summary: CI: BENCHMARK FAILED: org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
average latency is 5% worse than baseline.
                 Key: GEODE-7231
                 URL: https://issues.apache.org/jira/browse/GEODE-7231
             Project: Geode
          Issue Type: Bug
          Components: benchmarks, ci
            Reporter: Jinmei Liao


https://concourse.gemfire-ci.info/teams/main/pipelines/gemfire-develop-main/jobs/Benchmark/builds/106

> Task :harness:analyzeRun FAILED
Running analyzer
Comparing test result at /tmp/build/2ea77113/results/benchmarks-gemfire-develop-main-Benchmark-106-27727/branch/benchmarks_20190921000552
to baseline at /tmp/build/2ea77113/results/benchmarks-gemfire-develop-main-Benchmark-106-27727/baseline/benchmarks_20190920215707
org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
              average ops/second  Baseline:    171383.24  Test:    159507.38  Difference:
  -6.9%
       ops/second standard error  Baseline:       499.38  Test:       473.79  Difference:
  -5.1%
   ops/second standard deviation  Baseline:      8635.16  Test:      8192.61  Difference:
  -5.1%
      YS 99th percentile latency  Baseline:      1804.00  Test:      1903.00  Difference:
  +5.5%
                  median latency  Baseline:    637951.00  Test:    685055.00  Difference:
  +7.4%
         90th percentile latency  Baseline:   1198079.00  Test:   1320959.00  Difference:
 +10.3%
         99th percentile latency  Baseline:   4059135.00  Test:   4159487.00  Difference:
  +2.5%
       99.9th percentile latency  Baseline:  13574143.00  Test:  14589951.00  Difference:
  +7.5%
                 average latency  Baseline:    839055.57  Test:    901530.36  Difference:
  +7.4%
      latency standard deviation  Baseline:    929586.39  Test:    965788.36  Difference:
  +3.9%
          latency standard error  Baseline:       129.65  Test:       139.61  Difference:
  +7.7%
org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithFiltersBenchmark
              average ops/second  Baseline:    442946.14  Test:    442990.08  Difference:
  +0.0%
       ops/second standard error  Baseline:       435.26  Test:       418.38  Difference:
  -3.9%
   ops/second standard deviation  Baseline:      7526.29  Test:      7246.63  Difference:
  -3.7%
      YS 99th percentile latency  Baseline:      1504.00  Test:      1503.00  Difference:
  -0.1%
                  median latency  Baseline:    482559.00  Test:    483327.00  Difference:
  +0.2%
         90th percentile latency  Baseline:    988159.00  Test:    987135.00  Difference:
  -0.1%
         99th percentile latency  Baseline:   3557375.00  Test:   3532799.00  Difference:
  -0.7%
       99.9th percentile latency  Baseline:  10248191.00  Test:  10231807.00  Difference:
  -0.2%
                 average latency  Baseline:    649048.91  Test:    648977.69  Difference:
  -0.0%
      latency standard deviation  Baseline:    757279.93  Test:    758841.81  Difference:
  +0.2%
          latency standard error  Baseline:        65.70  Test:        65.83  Difference:
  +0.2%
org.apache.geode.benchmark.tests.PartitionedGetBenchmark
              average ops/second  Baseline:    825000.20  Test:    827836.08  Difference:
  +0.3%
       ops/second standard error  Baseline:       462.66  Test:       485.61  Difference:
  +5.0%
   ops/second standard deviation  Baseline:      8000.22  Test:      8411.04  Difference:
  +5.1%
      YS 99th percentile latency  Baseline:       904.00  Test:       905.00  Difference:
  +0.1%
                  median latency  Baseline:    357631.00  Test:    356607.00  Difference:
  -0.3%
         90th percentile latency  Baseline:    492287.00  Test:    489983.00  Difference:
  -0.5%
         99th percentile latency  Baseline:   2369535.00  Test:   2375679.00  Difference:
  +0.3%
       99.9th percentile latency  Baseline:   6729727.00  Test:   6565887.00  Difference:
  -2.4%
                 average latency  Baseline:    435231.89  Test:    433718.19  Difference:
  -0.3%
      latency standard deviation  Baseline:    535757.54  Test:    542177.55  Difference:
  +1.2%
          latency standard error  Baseline:        34.06  Test:        34.41  Difference:
  +1.0%
org.apache.geode.benchmark.tests.PartitionedGetLongBenchmark
              average ops/second  Baseline:   1038960.75  Test:   1039353.89  Difference:
  +0.0%
       ops/second standard error  Baseline:      1042.48  Test:       442.50  Difference:
 -57.6%
   ops/second standard deviation  Baseline:     18026.21  Test:      7651.57  Difference:
 -57.6%
      YS 99th percentile latency  Baseline:      1002.00  Test:      1002.00  Difference:
  +0.0%
                  median latency  Baseline:    305151.00  Test:    305151.00  Difference:
  +0.0%
         90th percentile latency  Baseline:    459519.00  Test:    459007.00  Difference:
  -0.1%
         99th percentile latency  Baseline:   1418239.00  Test:   1424383.00  Difference:
  +0.4%
       99.9th percentile latency  Baseline:   3729407.00  Test:   3788799.00  Difference:
  +1.6%
                 average latency  Baseline:    345199.75  Test:    345276.27  Difference:
  +0.0%
      latency standard deviation  Baseline:    364741.13  Test:    387501.17  Difference:
  +6.2%
          latency standard error  Baseline:        20.66  Test:        21.95  Difference:
  +6.2%
org.apache.geode.benchmark.tests.PartitionedIndexedQueryBenchmark
              average ops/second  Baseline:     18282.12  Test:     18247.08  Difference:
  -0.2%
       ops/second standard error  Baseline:         7.85  Test:        10.12  Difference:
 +28.9%
   ops/second standard deviation  Baseline:       135.73  Test:       174.91  Difference:
 +28.9%
      YS 99th percentile latency  Baseline:     20093.18  Test:     20092.77  Difference:
  -0.0%
                  median latency  Baseline:  14557183.00  Test:  13950975.00  Difference:
  -4.2%
         90th percentile latency  Baseline:  20594687.00  Test:  23232511.00  Difference:
 +12.8%
         99th percentile latency  Baseline:  46235647.00  Test:  60358655.00  Difference:
 +30.5%
       99.9th percentile latency  Baseline:  58621951.00  Test:  70582271.00  Difference:
 +20.4%
                 average latency  Baseline:  15752630.91  Test:  15779064.59  Difference:
  +0.2%
      latency standard deviation  Baseline:   7386025.34  Test:  10516174.80  Difference:
 +42.4%
          latency standard error  Baseline:      3154.86  Test:      4495.79  Difference:
 +42.5%
org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark
              average ops/second  Baseline:       654.62  Test:       649.58  Difference:
  -0.8%
       ops/second standard error  Baseline:         3.82  Test:         4.17  Difference:
  +9.4%
   ops/second standard deviation  Baseline:        65.09  Test:        71.08  Difference:
  +9.2%
      YS 99th percentile latency  Baseline:     20099.00  Test:     20099.00  Difference:
  +0.0%
                  median latency  Baseline:  67960831.00  Test:  69468159.00  Difference:
  +2.2%
         90th percentile latency  Baseline: 217448447.00  Test: 218890239.00  Difference:
  +0.7%
         99th percentile latency  Baseline: 312999935.00  Test: 316407807.00  Difference:
  +1.1%
       99.9th percentile latency  Baseline: 395837439.00  Test: 390332415.00  Difference:
  -1.4%
                 average latency  Baseline: 113043700.31  Test: 114136328.08  Difference:
  +1.0%
      latency standard deviation  Baseline:  77281720.43  Test:  77650547.89  Difference:
  +0.5%
          latency standard error  Baseline:    176865.03  Test:    178541.92  Difference:
  +0.9%
org.apache.geode.benchmark.tests.PartitionedPutAllLongBenchmark
              average ops/second  Baseline:       651.36  Test:       645.87  Difference:
  -0.8%
       ops/second standard error  Baseline:         4.14  Test:         4.15  Difference:
  +0.3%
   ops/second standard deviation  Baseline:        70.46  Test:        70.77  Difference:
  +0.4%
      YS 99th percentile latency  Baseline:     20099.00  Test:     20099.00  Difference:
  +0.0%
                  median latency  Baseline:  64454655.00  Test:  67895295.00  Difference:
  +5.3%
         90th percentile latency  Baseline: 225574911.00  Test: 220069887.00  Difference:
  -2.4%
         99th percentile latency  Baseline: 352583679.00  Test: 325844991.00  Difference:
  -7.6%
       99.9th percentile latency  Baseline: 427032575.00  Test: 413925375.00  Difference:
  -3.1%
                 average latency  Baseline: 113828250.36  Test: 114725443.16  Difference:
  +0.8%
      latency standard deviation  Baseline:  82646931.03  Test:  79661855.97  Difference:
  -3.6%
          latency standard error  Baseline:    189811.47  Test:    183679.48  Difference:
  -3.2%
org.apache.geode.benchmark.tests.PartitionedPutBenchmark
              average ops/second  Baseline:    419374.09  Test:    414255.28  Difference:
  -1.2%
       ops/second standard error  Baseline:       996.43  Test:       736.95  Difference:
 -26.0%
   ops/second standard deviation  Baseline:     17229.96  Test:     12743.12  Difference:
 -26.0%
      YS 99th percentile latency  Baseline:      1501.00  Test:      1502.00  Difference:
  +0.1%
                  median latency  Baseline:    676351.00  Test:    676351.00  Difference:
  +0.0%
         90th percentile latency  Baseline:    993279.00  Test:   1026047.00  Difference:
  +3.3%
         99th percentile latency  Baseline:   1644543.00  Test:   1753087.00  Difference:
  +6.6%
       99.9th percentile latency  Baseline:  66191359.00  Test:  65404927.00  Difference:
  -1.2%
                 average latency  Baseline:    857166.67  Test:    867866.53  Difference:
  +1.2%
      latency standard deviation  Baseline:   2941133.91  Test:   2966121.87  Difference:
  +0.8%
          latency standard error  Baseline:       262.23  Test:       266.09  Difference:
  +1.5%
org.apache.geode.benchmark.tests.PartitionedPutLongBenchmark
              average ops/second  Baseline:    411434.00  Test:    407384.65  Difference:
  -1.0%
       ops/second standard error  Baseline:       698.57  Test:       757.94  Difference:
  +8.5%
   ops/second standard deviation  Baseline:     12079.39  Test:     13127.90  Difference:
  +8.7%
      YS 99th percentile latency  Baseline:      1501.00  Test:      1502.00  Difference:
  +0.1%
                  median latency  Baseline:    673279.00  Test:    674815.00  Difference:
  +0.2%
         90th percentile latency  Baseline:   1010687.00  Test:   1026559.00  Difference:
  +1.6%
         99th percentile latency  Baseline:   1696767.00  Test:   1762303.00  Difference:
  +3.9%
       99.9th percentile latency  Baseline:  73400319.00  Test:  74842111.00  Difference:
  +2.0%
                 average latency  Baseline:    873796.73  Test:    882648.37  Difference:
  +1.0%
      latency standard deviation  Baseline:   3343859.26  Test:   3397106.08  Difference:
  +1.6%
          latency standard error  Baseline:       301.01  Test:       307.33  Difference:
  +2.1%
org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionBenchmark
              average ops/second  Baseline:    385497.82  Test:    383556.25  Difference:
  -0.5%
       ops/second standard error  Baseline:       157.81  Test:       187.92  Difference:
 +19.1%
   ops/second standard deviation  Baseline:      2728.75  Test:      3249.37  Difference:
 +19.1%
      YS 99th percentile latency  Baseline:       575.00  Test:       575.25  Difference:
  +0.0%
                  median latency  Baseline:    273919.00  Test:    275455.00  Difference:
  +0.6%
         90th percentile latency  Baseline:    368127.00  Test:    369663.00  Difference:
  +0.4%
         99th percentile latency  Baseline:    461055.00  Test:    471551.00  Difference:
  +2.3%
       99.9th percentile latency  Baseline:    886783.00  Test:    671743.00  Difference:
 -24.2%
                 average latency  Baseline:    279243.44  Test:    280609.57  Difference:
  +0.5%
      latency standard deviation  Baseline:    188613.48  Test:    186659.89  Difference:
  -1.0%
          latency standard error  Baseline:        17.54  Test:        17.40  Difference:
  -0.8%
org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithArgumentsBenchmark
              average ops/second  Baseline:    595408.61  Test:    615069.19  Difference:
  +3.3%
       ops/second standard error  Baseline:       549.38  Test:       402.08  Difference:
 -26.8%
   ops/second standard deviation  Baseline:      9499.75  Test:      6952.69  Difference:
 -26.8%
      YS 99th percentile latency  Baseline:      1505.00  Test:      1503.00  Difference:
  -0.1%
                  median latency  Baseline:    731647.00  Test:    775679.00  Difference:
  +6.0%
         90th percentile latency  Baseline:   1094655.00  Test:   1083391.00  Difference:
  -1.0%
         99th percentile latency  Baseline:   9043967.00  Test:   6017023.00  Difference:
 -33.5%
       99.9th percentile latency  Baseline:  22757375.00  Test:  23691263.00  Difference:
  +4.1%
                 average latency  Baseline:    965601.30  Test:    934625.03  Difference:
  -3.2%
      latency standard deviation  Baseline:   1702556.73  Test:   1590548.30  Difference:
  -6.6%
          latency standard error  Baseline:       127.41  Test:       117.11  Difference:
  -8.1%
org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithFiltersBenchmark
              average ops/second  Baseline:    584622.61  Test:    583232.58  Difference:
  -0.2%
       ops/second standard error  Baseline:       369.33  Test:       377.86  Difference:
  +2.3%
   ops/second standard deviation  Baseline:      6386.34  Test:      6533.81  Difference:
  +2.3%
      YS 99th percentile latency  Baseline:      1103.00  Test:      1103.00  Difference:
  +0.0%
                  median latency  Baseline:    496639.00  Test:    520191.00  Difference:
  +4.7%
         90th percentile latency  Baseline:    731135.00  Test:    736255.00  Difference:
  +0.7%
         99th percentile latency  Baseline:   3874815.00  Test:   3444735.00  Difference:
 -11.1%
       99.9th percentile latency  Baseline:  11780095.00  Test:  10878975.00  Difference:
  -7.6%
                 average latency  Baseline:    614285.67  Test:    615818.03  Difference:
  +0.2%
      latency standard deviation  Baseline:    821732.66  Test:    723485.75  Difference:
 -12.0%
          latency standard error  Baseline:        62.05  Test:        54.70  Difference:
 -11.9%
org.apache.geode.benchmark.tests.ReplicatedGetBenchmark
              average ops/second  Baseline:    846698.07  Test:    846876.85  Difference:
  +0.0%
       ops/second standard error  Baseline:       455.80  Test:       398.00  Difference:
 -12.7%
   ops/second standard deviation  Baseline:      7881.58  Test:      6893.61  Difference:
 -12.5%
      YS 99th percentile latency  Baseline:       905.00  Test:       905.00  Difference:
  +0.0%
                  median latency  Baseline:    349183.00  Test:    348415.00  Difference:
  -0.2%
         90th percentile latency  Baseline:    484095.00  Test:    486655.00  Difference:
  +0.5%
         99th percentile latency  Baseline:   2435071.00  Test:   2369535.00  Difference:
  -2.7%
       99.9th percentile latency  Baseline:   6770687.00  Test:   6574079.00  Difference:
  -2.9%
                 average latency  Baseline:    424071.21  Test:    423929.31  Difference:
  -0.0%
      latency standard deviation  Baseline:    530295.63  Test:    521763.35  Difference:
  -1.6%
          latency standard error  Baseline:        33.28  Test:        32.74  Difference:
  -1.6%
org.apache.geode.benchmark.tests.ReplicatedGetLongBenchmark
              average ops/second  Baseline:   1056863.08  Test:   1056316.60  Difference:
  -0.1%
       ops/second standard error  Baseline:       476.01  Test:       533.16  Difference:
 +12.0%
   ops/second standard deviation  Baseline:      8231.02  Test:      9219.21  Difference:
 +12.0%
      YS 99th percentile latency  Baseline:      1002.00  Test:      1001.00  Difference:
  -0.1%
                  median latency  Baseline:    299775.00  Test:    299775.00  Difference:
  +0.0%
         90th percentile latency  Baseline:    451071.00  Test:    448255.00  Difference:
  -0.6%
         99th percentile latency  Baseline:   1371135.00  Test:   1392639.00  Difference:
  +1.6%
       99.9th percentile latency  Baseline:   3262463.00  Test:   3409919.00  Difference:
  +4.5%
                 average latency  Baseline:    339496.49  Test:    339654.77  Difference:
  +0.0%
      latency standard deviation  Baseline:    342396.65  Test:    373899.72  Difference:
  +9.2%
          latency standard error  Baseline:        19.23  Test:        21.01  Difference:
  +9.2%
org.apache.geode.benchmark.tests.ReplicatedIndexedQueryBenchmark
              average ops/second  Baseline:     18229.31  Test:     18224.52  Difference:
  -0.0%
       ops/second standard error  Baseline:        10.16  Test:         8.23  Difference:
 -19.0%
   ops/second standard deviation  Baseline:       175.60  Test:       142.52  Difference:
 -18.8%
      YS 99th percentile latency  Baseline:     20093.73  Test:     20093.00  Difference:
  -0.0%
                  median latency  Baseline:  12836863.00  Test:  13041663.00  Difference:
  +1.6%
         90th percentile latency  Baseline:  26427391.00  Test:  25903103.00  Difference:
  -2.0%
         99th percentile latency  Baseline:  71172095.00  Test:  69992447.00  Difference:
  -1.7%
       99.9th percentile latency  Baseline:  83427327.00  Test:  82182143.00  Difference:
  -1.5%
                 average latency  Baseline:  15800858.04  Test:  15802259.29  Difference:
  +0.0%
      latency standard deviation  Baseline:  13064421.06  Test:  12769643.21  Difference:
  -2.3%
          latency standard error  Baseline:      5588.98  Test:      5463.10  Difference:
  -2.3%
org.apache.geode.benchmark.tests.ReplicatedPutAllBenchmark
              average ops/second  Baseline:       686.64  Test:       699.59  Difference:
  +1.9%
       ops/second standard error  Baseline:         4.27  Test:         4.15  Difference:
  -2.8%
   ops/second standard deviation  Baseline:        73.34  Test:        71.06  Difference:
  -3.1%
      YS 99th percentile latency  Baseline:     20099.00  Test:     20099.00  Difference:
  +0.0%
                  median latency  Baseline:  77856767.00  Test:  73531391.00  Difference:
  -5.6%
         90th percentile latency  Baseline: 204079103.00  Test: 202899455.00  Difference:
  -0.6%
         99th percentile latency  Baseline: 330563583.00  Test: 324534271.00  Difference:
  -1.8%
       99.9th percentile latency  Baseline: 408682495.00  Test: 369098751.00  Difference:
  -9.7%
                 average latency  Baseline: 106331049.47  Test: 104780049.58  Difference:
  -1.5%
      latency standard deviation  Baseline:  71143931.88  Test:  70631162.82  Difference:
  -0.7%
          latency standard error  Baseline:    157915.25  Test:    155630.00  Difference:
  -1.4%
org.apache.geode.benchmark.tests.ReplicatedPutAllLongBenchmark
              average ops/second  Baseline:       684.32  Test:       701.87  Difference:
  +2.6%
       ops/second standard error  Baseline:         4.09  Test:         4.03  Difference:
  -1.4%
   ops/second standard deviation  Baseline:        70.20  Test:        69.19  Difference:
  -1.4%
      YS 99th percentile latency  Baseline:     20099.00  Test:     20099.00  Difference:
  +0.0%
                  median latency  Baseline:  74579967.00  Test:  72941567.00  Difference:
  -2.2%
         90th percentile latency  Baseline: 207880191.00  Test: 203554815.00  Difference:
  -2.1%
         99th percentile latency  Baseline: 344719359.00  Test: 333447167.00  Difference:
  -3.3%
       99.9th percentile latency  Baseline: 392691711.00  Test: 376176639.00  Difference:
  -4.2%
                 average latency  Baseline: 106559160.43  Test: 104315677.10  Difference:
  -2.1%
      latency standard deviation  Baseline:  73550701.86  Test:  71595613.07  Difference:
  -2.7%
          latency standard error  Baseline:    163397.60  Test:    157398.24  Difference:
  -3.7%
org.apache.geode.benchmark.tests.ReplicatedPutBenchmark
              average ops/second  Baseline:    439483.79  Test:    434437.21  Difference:
  -1.1%
       ops/second standard error  Baseline:       819.57  Test:       746.63  Difference:
  -8.9%
   ops/second standard deviation  Baseline:     14171.70  Test:     12910.43  Difference:
  -8.9%
      YS 99th percentile latency  Baseline:      1302.00  Test:      1304.00  Difference:
  +0.2%
                  median latency  Baseline:    662015.00  Test:    662015.00  Difference:
  +0.0%
         90th percentile latency  Baseline:    894463.00  Test:    925183.00  Difference:
  +3.4%
         99th percentile latency  Baseline:   1791999.00  Test:   1894399.00  Difference:
  +5.7%
       99.9th percentile latency  Baseline:  69730303.00  Test:  68091903.00  Difference:
  -2.3%
                 average latency  Baseline:    818083.55  Test:    826988.90  Difference:
  +1.1%
      latency standard deviation  Baseline:   2889557.20  Test:   2861827.56  Difference:
  -1.0%
          latency standard error  Baseline:       251.67  Test:       250.67  Difference:
  -0.4%
org.apache.geode.benchmark.tests.ReplicatedPutLongBenchmark
              average ops/second  Baseline:    430225.52  Test:    422842.19  Difference:
  -1.7%
       ops/second standard error  Baseline:       808.44  Test:       736.25  Difference:
  -8.9%
   ops/second standard deviation  Baseline:     13979.30  Test:     12730.95  Difference:
  -8.9%
      YS 99th percentile latency  Baseline:      1303.00  Test:      1303.00  Difference:
  +0.0%
                  median latency  Baseline:    662527.00  Test:    664063.00  Difference:
  +0.2%
         90th percentile latency  Baseline:    912383.00  Test:    931839.00  Difference:
  +2.1%
         99th percentile latency  Baseline:   1898495.00  Test:   1918975.00  Difference:
  +1.1%
       99.9th percentile latency  Baseline:  73138175.00  Test:  80674815.00  Difference:
 +10.3%
                 average latency  Baseline:    835773.93  Test:    850337.27  Difference:
  +1.7%
      latency standard deviation  Baseline:   3198061.99  Test:   3392288.63  Difference:
  +6.1%
          latency standard error  Baseline:       281.54  Test:       301.23  Difference:
  +7.0%
BENCHMARK FAILED: org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
average latency is 5% worse than baseline.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message