kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a...@apache.org
Subject [2/2] kudu git commit: transaction_tracker: add more information to the out-of-memory message
Date Wed, 24 Oct 2018 17:55:04 GMT
transaction_tracker: add more information to the out-of-memory message

A Slack user reported seeing an error like this:

  transaction_tracker.cc:119] Transaction failed, tablet \
    00000000000000000000000000000000 transaction memory consumption (0) has \
    exceeded its limit (67108864) or the limit of an ancestral tracker

This doesn't make sense; how could the limit be exceeded if the consumption
is 0? As far as I can tell, it can only happen if the failed transaction had
a colossal footprint of its own.

This patch changes the generated error to include the footprint of the
transaction itself.

Change-Id: Ic4017c1aecec29cd4b4b04c15efa4bf38c925cb9
Reviewed-on: http://gerrit.cloudera.org:8080/11765
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Andrew Wong <awong@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/fff0da84
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/fff0da84
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/fff0da84

Branch: refs/heads/master
Commit: fff0da8408abc70b192c303085f3e45175d495a6
Parents: d1f8c23
Author: Adar Dembo <adar@cloudera.com>
Authored: Wed Oct 24 09:38:01 2018 -0700
Committer: Adar Dembo <adar@cloudera.com>
Committed: Wed Oct 24 17:54:40 2018 +0000

----------------------------------------------------------------------
 .../tablet/transactions/transaction_tracker-test.cc   | 14 ++++++++------
 src/kudu/tablet/transactions/transaction_tracker.cc   |  7 ++++---
 2 files changed, 12 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/fff0da84/src/kudu/tablet/transactions/transaction_tracker-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/transactions/transaction_tracker-test.cc b/src/kudu/tablet/transactions/transaction_tracker-test.cc
index dbd4e07..44d21c0 100644
--- a/src/kudu/tablet/transactions/transaction_tracker-test.cc
+++ b/src/kudu/tablet/transactions/transaction_tracker-test.cc
@@ -38,6 +38,7 @@
 #include "kudu/util/mem_tracker.h"
 #include "kudu/util/metrics.h"
 #include "kudu/util/monotime.h"
+#include "kudu/util/scoped_cleanup.h"
 #include "kudu/util/status.h"
 #include "kudu/util/test_macros.h"
 #include "kudu/util/test_util.h"
@@ -240,14 +241,19 @@ TEST_F(TransactionTrackerTest, TestTooManyTransactions) {
   // carries an empty ReplicateMsg), so we'll just add as many as possible
   // and check that when we fail, it's because we've hit the limit.
   Status s;
-  vector<scoped_refptr<TransactionDriver> > drivers;
+  vector<scoped_refptr<TransactionDriver>> drivers;
+  SCOPED_CLEANUP({
+    for (const auto& d : drivers) {
+      d->Abort(Status::Aborted(""));
+    }
+  });
   for (int i = 0; s.ok();i++) {
     s = AddDrivers(1, &drivers);
   }
 
   LOG(INFO) << "Added " << drivers.size() << " drivers";
   ASSERT_TRUE(s.IsServiceUnavailable());
-  ASSERT_STR_CONTAINS(s.ToString(), "exceeded its limit");
+  ASSERT_STR_CONTAINS(s.ToString(), "exceeded its transaction memory limit");
   NO_FATALS(CheckMetrics(entity_, drivers.size(), 0, 1));
   NO_FATALS(CheckMemTracker(t));
 
@@ -262,10 +268,6 @@ TEST_F(TransactionTrackerTest, TestTooManyTransactions) {
   ASSERT_OK(AddDrivers(1, &drivers));
   NO_FATALS(CheckMemTracker(t));
 
-  // Clean up.
-  for (const scoped_refptr<TransactionDriver>& driver : drivers) {
-    driver->Abort(Status::Aborted(""));
-  }
 }
 
 } // namespace tablet

http://git-wip-us.apache.org/repos/asf/kudu/blob/fff0da84/src/kudu/tablet/transactions/transaction_tracker.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/transactions/transaction_tracker.cc b/src/kudu/tablet/transactions/transaction_tracker.cc
index cc459ec..34a348c 100644
--- a/src/kudu/tablet/transactions/transaction_tracker.cc
+++ b/src/kudu/tablet/transactions/transaction_tracker.cc
@@ -111,10 +111,11 @@ Status TransactionTracker::Add(TransactionDriver* driver) {
     TabletReplica* replica = driver->state()->tablet_replica();
 
     string msg = Substitute(
-        "Transaction failed, tablet $0 transaction memory consumption ($1) "
-        "has exceeded its limit ($2) or the limit of an ancestral tracker",
+        "Transaction failed, tablet $0 exceeded its transaction memory limit "
+        "($1) or the limit of an ancestral memory tracker. This transaction "
+        "consumption's was $2 and the current consumption is $3.",
         replica ? replica->tablet()->tablet_id() : "(unknown)",
-        mem_tracker_->consumption(), mem_tracker_->limit());
+        mem_tracker_->limit(), driver_mem_footprint, mem_tracker_->consumption());
 
     KLOG_EVERY_N_SECS(WARNING, 1) << msg << THROTTLE_MSG;
 


Mime
View raw message