kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject [1/4] kudu git commit: KUDU-1949. Maintenance Manager should trigger flushes earlier
Date Fri, 05 May 2017 20:49:24 GMT
Repository: kudu
Updated Branches:
  refs/heads/master 46a886cd2 -> 4df9f5969


KUDU-1949. Maintenance Manager should trigger flushes earlier

This changes the maintenance manager to start triggering flushes at the
60% heap usage threshold, and adjusts the memory limiting to only start
rejecting writes at the 80% threshold.

This is based on analysis in [1] ("After tweaking memory backpressure
rejection") in which we found that earlier flushing could keep the
system from making too many rejections.

I ran tpch_real_world with SF=300 as described in the doc below before
and after the patch. Some key metrics:

                Before       After
Mem rejections  1,162,527    62,790    (18.5x fewer)
Wall time       3156s        2490s     (1.26x faster)
Server CPU      37,348s      72,873s   (1.95x less efficient)

This matches the results from the original experiment. Because the
writes come in faster, we don't get enough time to run compactions, and
so each write ends up taking more CPU. Some future optimizations are
still pending which will reduce the CPU time per bloom lookup.
Regardless, this new behavior is configurable such that, if overall CPU
efficiency is more important than wall time, we could bump the "memory
pressure threshold" up higher than the "memory rejection threshold".

[1] https://docs.google.com/document/d/1U1IXS1XD2erZyq8_qG81A1gZaCeHcq2i0unea_eEf5c/edit#heading=h.gw87heq4fq6l

Change-Id: I8b7e703c82ac14fbce3a699bdf6a2f0fb4ed93a1
Reviewed-on: http://gerrit.cloudera.org:8080/6802
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: David Ribeiro Alves <davidralves@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/da2055f6
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/da2055f6
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/da2055f6

Branch: refs/heads/master
Commit: da2055f64d9ed717be6a3ca62868964aa54dc57d
Parents: 46a886c
Author: Todd Lipcon <todd@apache.org>
Authored: Thu May 4 13:42:08 2017 -0700
Committer: Todd Lipcon <todd@apache.org>
Committed: Fri May 5 18:26:56 2017 +0000

----------------------------------------------------------------------
 src/kudu/util/maintenance_manager.cc |  2 +-
 src/kudu/util/process_memory.cc      | 24 +++++++++++++++++++++++-
 src/kudu/util/process_memory.h       |  7 +++++++
 3 files changed, 31 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/da2055f6/src/kudu/util/maintenance_manager.cc
----------------------------------------------------------------------
diff --git a/src/kudu/util/maintenance_manager.cc b/src/kudu/util/maintenance_manager.cc
index d2206dc..387e79f 100644
--- a/src/kudu/util/maintenance_manager.cc
+++ b/src/kudu/util/maintenance_manager.cc
@@ -128,7 +128,7 @@ MaintenanceManager::MaintenanceManager(const Options& options)
           options.polling_interval_ms),
     completed_ops_count_(0),
     rand_(GetRandomSeed32()),
-    memory_pressure_func_(&process_memory::SoftLimitExceeded) {
+    memory_pressure_func_(&process_memory::UnderMemoryPressure) {
   CHECK_OK(ThreadPoolBuilder("MaintenanceMgr").set_min_threads(num_threads_)
                .set_max_threads(num_threads_).Build(&thread_pool_));
   uint32_t history_size = options.history_size == 0 ?

http://git-wip-us.apache.org/repos/asf/kudu/blob/da2055f6/src/kudu/util/process_memory.cc
----------------------------------------------------------------------
diff --git a/src/kudu/util/process_memory.cc b/src/kudu/util/process_memory.cc
index c4fb8d7..cac1e14 100644
--- a/src/kudu/util/process_memory.cc
+++ b/src/kudu/util/process_memory.cc
@@ -37,7 +37,12 @@ DEFINE_int64(memory_limit_hard_bytes, 0,
              "A value of -1 disables all memory limiting.");
 TAG_FLAG(memory_limit_hard_bytes, stable);
 
-DEFINE_int32(memory_limit_soft_percentage, 60,
+DEFINE_int32(memory_pressure_percentage, 60,
+             "Percentage of the hard memory limit that this daemon may "
+             "consume before flushing of in-memory data becomes prioritized.");
+TAG_FLAG(memory_pressure_percentage, advanced);
+
+DEFINE_int32(memory_limit_soft_percentage, 80,
              "Percentage of the hard memory limit that this daemon may "
              "consume before memory throttling of writes begins. The greater "
              "the excess, the higher the chance of throttling. In general, a "
@@ -65,6 +70,7 @@ namespace process_memory {
 namespace {
 int64_t g_hard_limit;
 int64_t g_soft_limit;
+int64_t g_pressure_threshold;
 
 ThreadSafeRandom* g_rand = nullptr;
 
@@ -163,6 +169,7 @@ void DoInitLimits() {
   }
   g_hard_limit = limit;
   g_soft_limit = FLAGS_memory_limit_soft_percentage * g_hard_limit / 100;
+  g_pressure_threshold = FLAGS_memory_pressure_percentage * g_hard_limit / 100;
 
   g_rand = new ThreadSafeRandom(1);
 
@@ -171,6 +178,9 @@ void DoInitLimits() {
   LOG(INFO) << StringPrintf("Process soft memory limit is %.6f GB",
                             (static_cast<float>(g_soft_limit) /
                              (1024.0 * 1024.0 * 1024.0)));
+  LOG(INFO) << StringPrintf("Process memory pressure threshold is %.6f GB",
+                            (static_cast<float>(g_pressure_threshold) /
+                             (1024.0 * 1024.0 * 1024.0)));
 }
 
 void InitLimits() {
@@ -223,6 +233,18 @@ int64_t HardLimit() {
   return g_hard_limit;
 }
 
+bool UnderMemoryPressure(double* current_capacity_pct) {
+  InitLimits();
+  int64_t consumption = CurrentConsumption();
+  if (consumption < g_pressure_threshold) {
+    return false;
+  }
+  if (current_capacity_pct) {
+    *current_capacity_pct = static_cast<double>(consumption) / g_hard_limit * 100;
+  }
+  return true;
+}
+
 bool SoftLimitExceeded(double* current_capacity_pct) {
   InitLimits();
   int64_t consumption = CurrentConsumption();

http://git-wip-us.apache.org/repos/asf/kudu/blob/da2055f6/src/kudu/util/process_memory.h
----------------------------------------------------------------------
diff --git a/src/kudu/util/process_memory.h b/src/kudu/util/process_memory.h
index 7b09f8b..1545dc7 100644
--- a/src/kudu/util/process_memory.h
+++ b/src/kudu/util/process_memory.h
@@ -27,6 +27,13 @@ namespace process_memory {
 // of the hard limit consumed is written to it.
 bool SoftLimitExceeded(double* current_capacity_pct);
 
+// Return true if we are under memory pressure (i.e if we are nearing the point at which
+// SoftLimitExceeded will begin to return true).
+//
+// If the process is under memory pressure, and 'current_capacity_pct' is not NULL,
+// the percentage of the hard limit consumed is written to it.
+bool UnderMemoryPressure(double* current_capacity_pct);
+
 // Potentially trigger a call to release tcmalloc memory back to the
 // OS, after the given amount of memory was released.
 void MaybeGCAfterRelease(int64_t released_bytes);


Mime
View raw message