kudu-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Serbin (Code Review)" <ger...@cloudera.org>
Subject [kudu-CR] [master] cache for table locations
Date Wed, 01 Jul 2020 06:29:44 GMT
Hello Kudu Jenkins, Andrew Wong, Grant Henke, Greg Solovyev, Bankim Bhavsar, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15971

to look at the new patch set (#9).

Change subject: [master] cache for table locations
......................................................................

[master] cache for table locations

This patch introduces a cache for table locations in catalog manager.

When running with 48 concurrent client threads, the performance of
CatalogManager::GetTableLocations() method improved about 100%
when the cache is enabled.
A smaller 14% improvement is observed for GetTableLocations RPC
when running with the same number of concurrent client threads and cache
enabled.  The test results are below.

I'm planning to add these test scenarios into kudu/scripts/benchmarks.sh
in a follow up patch.

========================================================================

After this patch with 128MByte cache enabled:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
  --benchmark_num_threads=48 \
  --table_locations_cache_capacity_mb=128

  GetTableLocations function call: 504187.6 req/sec

Before this patch:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
  --benchmark_num_threads=48

  GetTableLocations function call: 252443.4 req/sec

========================================================================

After this patch with 128MByte cache enabled:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
  --rpc_num_service_threads=32 \
  --benchmark_num_threads=48 \
  --table_locations_cache_capacity_mb=128

  GetTableLocations RPC: 40033.4 req/sec
  Stats on GetTableLocations RPC (times in microseconds):
  Count: 200167
  Mean: 155.144
  Percentiles:
     0%  (min) = 45
    25%        = 125
    50%  (med) = 147
    75%        = 177
    95%        = 223
    99%        = 314
    99.9%      = 652
    99.99%     = 1928
    100% (max) = 5096

Before this patch:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
  --rpc_num_service_threads=32 \
  --benchmark_num_threads=48

  GetTableLocations RPC: 34981 req/sec
  Stats on GetTableLocations RPC (times in microseconds):
  Count: 174905
  Mean: 249.308
  Percentiles:
     0%  (min) = 64
    25%        = 197
    50%  (med) = 231
    75%        = 284
    95%        = 386
    99%        = 556
    99.9%      = 980
    99.99%     = 2400
    100% (max) = 6113

Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
---
M src/kudu/integration-tests/table_locations-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master_service.cc
A src/kudu/master/table_locations_cache.cc
A src/kudu/master/table_locations_cache.h
A src/kudu/master/table_locations_cache_metrics.cc
A src/kudu/master/table_locations_cache_metrics.h
9 files changed, 955 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/71/15971/9
-- 
To view, visit http://gerrit.cloudera.org:8080/15971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
Gerrit-Change-Number: 15971
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <aserbin@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aserbin@cloudera.com>
Gerrit-Reviewer: Andrew Wong <awong@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <bankim@cloudera.com>
Gerrit-Reviewer: Grant Henke <granthenke@apache.org>
Gerrit-Reviewer: Greg Solovyev <gsolovyev@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <todd@apache.org>

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message