kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From granthe...@apache.org
Subject [1/3] kudu git commit: tablet_server-test: wait for leader in some tests
Date Thu, 18 Oct 2018 21:20:12 GMT
Repository: kudu
Updated Branches:
  refs/heads/master 53e028798 -> 785490ce5


tablet_server-test: wait for leader in some tests

The fix for KUDU-2463 made successful Raft elections a prerequisite for
scanning empty tablets. Under enough stress, even in a single replica
case, this may not complete immediately, and can thus, lead scans to
return a different error than expected. This patch updates the test
fixture that creates an empty tablet in tablet_server-test to wait until
this has passed before deeming the tablet ready for business.

The following test incantation:
tablet_server-test --gtest_filter=*BadProjectionTypes* --stress_cpu_threads=100

would fail fairly consistently without this patch.

Change-Id: Ie6ca079353d4d44a48defc386d94b4bda452ec1b
Reviewed-on: http://gerrit.cloudera.org:8080/11718
Tested-by: Andrew Wong <awong@cloudera.com>
Reviewed-by: Adar Dembo <adar@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/64d5fe52
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/64d5fe52
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/64d5fe52

Branch: refs/heads/master
Commit: 64d5fe5267d3f936e9b366d6fa3de8e5a7a42212
Parents: 53e0287
Author: Andrew Wong <awong@cloudera.com>
Authored: Wed Oct 17 16:20:38 2018 -0700
Committer: Andrew Wong <awong@cloudera.com>
Committed: Thu Oct 18 18:47:50 2018 +0000

----------------------------------------------------------------------
 src/kudu/tserver/tablet_server-test-base.cc | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/64d5fe52/src/kudu/tserver/tablet_server-test-base.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tserver/tablet_server-test-base.cc b/src/kudu/tserver/tablet_server-test-base.cc
index df68bbd..924bf5e 100644
--- a/src/kudu/tserver/tablet_server-test-base.cc
+++ b/src/kudu/tserver/tablet_server-test-base.cc
@@ -44,6 +44,7 @@
 #include "kudu/rpc/messenger.h"
 #include "kudu/rpc/rpc_controller.h"
 #include "kudu/tablet/local_tablet_writer.h"
+#include "kudu/tablet/mvcc.h"
 #include "kudu/tablet/tablet.h"
 #include "kudu/tablet/tablet_replica.h"
 #include "kudu/tserver/mini_tablet_server.h"
@@ -133,10 +134,22 @@ void TabletServerTestBase::StartTabletServer(int num_data_dirs) {
 Status TabletServerTestBase::WaitForTabletRunning(const char *tablet_id) {
   scoped_refptr<tablet::TabletReplica> tablet_replica;
   const auto* tablet_manager = mini_server_->server()->tablet_manager();
+  const auto kTimeout = MonoDelta::FromSeconds(10);
   RETURN_NOT_OK(tablet_manager->GetTabletReplica(tablet_id, &tablet_replica));
-  RETURN_NOT_OK(tablet_replica->WaitUntilConsensusRunning(MonoDelta::FromSeconds(10)));
+  RETURN_NOT_OK(tablet_replica->WaitUntilConsensusRunning(kTimeout));
   RETURN_NOT_OK(
-      tablet_replica->consensus()->WaitUntilLeaderForTests(MonoDelta::FromSeconds(10)));
+      tablet_replica->consensus()->WaitUntilLeaderForTests(kTimeout));
+
+  // KUDU-2463: Even though the tablet thinks its leader, for correctness, it
+  // must wait to finish replicating its no-op (even as a single replica)
+  // before being available to scans.
+  MonoTime deadline = MonoTime::Now() + kTimeout;
+  while (!tablet_replica->tablet()->mvcc_manager()->CheckIsSafeTimeInitialized().ok())
{
+    if (MonoTime::Now() >= deadline) {
+      return Status::TimedOut("mvcc did not advance safe time within timeout");
+    }
+    SleepFor(MonoDelta::FromMilliseconds(10));
+  }
 
   // KUDU-2444: Even though the tablet replica is fully running, the tablet
   // manager may regard it as still transitioning to the running state.


Mime
View raw message