kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From granthe...@apache.org
Subject [2/6] kudu git commit: benchmarks: update parameters for mt-bloomfile benchmark
Date Mon, 30 Apr 2018 21:10:17 GMT
benchmarks: update parameters for mt-bloomfile benchmark

The old parameters were somewhat unrealistic:
- The default bloom block size is 4kb, but it was using 32kb.
- The cache capacity was set to 1MB, which meant that each shard on a
  multicore system was only 50KB or less. With the new optimization that
  allows 1% slack in MemTracker accounting, that slack was only
  500bytes or less. With 4kb bloom blocks, that optimization was
  effectively disabled.

The new parameters are slightly more realistic while keeping the
performance reasonable: 100MB cache capacity (>1MB per shard even on 88
core system) and 4kb blocks. Runtime is still reasonable - 4-5 seconds
for the query phase. The number of unique keys is bumped up to 150M to
ensure cache churn.

Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Reviewed-on: http://gerrit.cloudera.org:8080/10242
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins

Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/91730158
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/91730158
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/91730158

Branch: refs/heads/master
Commit: 9173015808074c6f771d08be25812446f6a87d48
Parents: 60acdb9
Author: Todd Lipcon <todd@apache.org>
Authored: Mon Apr 30 10:21:04 2018 -0700
Committer: Todd Lipcon <todd@apache.org>
Committed: Mon Apr 30 18:58:22 2018 +0000

 src/kudu/scripts/benchmarks.sh | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/kudu/scripts/benchmarks.sh b/src/kudu/scripts/benchmarks.sh
index 1c9f18c..346debe 100755
--- a/src/kudu/scripts/benchmarks.sh
+++ b/src/kudu/scripts/benchmarks.sh
@@ -208,10 +208,12 @@ run_benchmarks() {
   # run mt-bloomfile-test 5 times. 20-30 seconds per run.
-  # The block cache is set to 1MB to generate churn.
+  # We use 150M keys so the total bloom space is >150MB, and then
+  # set the cache capacity to be only 100M to ensure that the cache
+  # churns a lot.
   for i in $(seq 1 $NUM_SAMPLES); do
-    ./build/latest/bin/mt-bloomfile-test --benchmark_queries=2000000 --bloom_size_bytes=32768
-      --n_keys=5000000 --block_cache_capacity_mb=1 &> $LOGDIR/$MT_BLOOM_TEST$i.log
+    ./build/latest/bin/mt-bloomfile-test --benchmark_queries=2000000 --bloom_size_bytes=4096
+      --n_keys=150000000 --block_cache_capacity_mb=100 &> $LOGDIR/$MT_BLOOM_TEST$i.log
   # run wire_protocol-test 5 times. 6 seconds per run

View raw message