kudu-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Serbin (Code Review)" <ger...@cloudera.org>
Subject [kudu-CR] [master] cache for table locations
Date Tue, 30 Jun 2020 06:17:35 GMT
Hello Kudu Jenkins, Andrew Wong, Grant Henke, Bankim Bhavsar, Todd Lipcon, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15971

to look at the new patch set (#7).

Change subject: [master] cache for table locations
......................................................................

[master] cache for table locations

This patch introduces a cache for table locations in catalog manager.

When running with 48 concurrent client threads, the performance of
CatalogManager::GetTableLocations() method improved about 130%
when the cache is enabled.
A smaller 12% improvement is observed for GetTableLocations RPC
when running with the same number of concurrent client threads and cache
enabled.  The test results are below.

I'm planning to add these test scenarios into kudu/scripts/benchmarks.sh
in a follow up patch.

========================================================================

After this patch with 128MByte cache enabled:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
  --benchmark_num_threads=48 \
  --table_locations_cache_capacity_mb=128

  GetTableLocations function call: 588056.2 req/sec

Before this patch:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
  --benchmark_num_threads=48

  GetTableLocations function call: 252443.4 req/sec

========================================================================

After this patch with 128MByte cache enabled:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
  --rpc_num_service_threads=24 \
  --benchmark_num_threads=48 \
  --table_locations_cache_capacity_mb=128

  GetTableLocations RPC: 39328.6 req/sec
  Stats on GetTableLocations RPC (times in microseconds):
  Count: 196643
  Mean: 149.526
  Percentiles:
     0%  (min) = 40
    25%        = 120
    50%  (med) = 142
    75%        = 172
    95%        = 216
    99%        = 308
    99.9%      = 652
    99.99%     = 1424
    100% (max) = 8480

Before this patch:

table_locations-itest \
  --gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
  --rpc_num_service_threads=24 \
  --benchmark_num_threads=48

  GetTableLocations RPC: 34833.2 req/sec
  Stats on GetTableLocations RPC (times in microseconds):
  Count: 174166
  Mean: 241.475
  Percentiles:
     0%  (min) = 61
    25%        = 189
    50%  (med) = 225
    75%        = 278
    95%        = 374
    99%        = 532
    99.9%      = 956
    99.99%     = 1848
    100% (max) = 5834

Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
---
M src/kudu/integration-tests/table_locations-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master_service.cc
A src/kudu/master/table_locations_cache.cc
A src/kudu/master/table_locations_cache.h
A src/kudu/master/table_locations_cache_metrics.cc
A src/kudu/master/table_locations_cache_metrics.h
9 files changed, 941 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/71/15971/7
-- 
To view, visit http://gerrit.cloudera.org:8080/15971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
Gerrit-Change-Number: 15971
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <aserbin@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aserbin@cloudera.com>
Gerrit-Reviewer: Andrew Wong <awong@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <bankim@cloudera.com>
Gerrit-Reviewer: Grant Henke <granthenke@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <todd@apache.org>

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message