kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jdcry...@apache.org
Subject [1/2] kudu git commit: KUDU-1607. Unpin tablet flush after failed bootstrap
Date Tue, 18 Apr 2017 21:24:20 GMT
Repository: kudu
Updated Branches:
  refs/heads/branch-1.2.x 50039911b -> b22bd3358


KUDU-1607. Unpin tablet flush after failed bootstrap

We have heard reports that, in certain scenarios, a failed tablet is
unable to be deleted. After some investigation, I determined that this
is because we neglect to unpin the Tablet when tablet bootstrap fails.

When a table is being deleted, we delete each tablet superblock by
calling TabletMetadata::DeleteSuperBlock(). This method double-checks
that no orphaned blocks remain referenced to ensure we don't leak disk
space. That requires TabletMetadata::DeleteTabletData() to be called
first, which invokes Flush() twice to ensure that no orphaned blocks are
referenced on disk upon returning. However, if the tablet is pinned,
Flush() silently becomes a no-op (except for a log message that is
printed) and so DeleteSuperBlock() also fails, resulting in the tablet
not being fully deleted and the following warning message being written
to the log file:

W0317 13:05:19.324373 13242 ts_tablet_manager.cc:634] Invalid argument: Unable to delete on-disk
data from tablet d1b857ddaa0743c79c9bcbd9b2fda174: The metadata for tablet d1b857ddaa0743c79c9bcbd9b2fda174
still references orphaned blocks. Call DeleteTabletData() first

This patch makes the following changes:

1. Always unpin the tablet after pinning it.
2. Add a new itest that fails without that change, reproducing the
   above warning message.
3. Add a little more test infra to MiniCluster (and the MiniClusterBase
   interface) to make it easier to use the helper functions in
   cluster_itest_util.h in MiniCluster-based itests.

Note: This backport has been modified not to rely on KUDU-1741.

Change-Id: Id274c6ee1da75bc6f92ab91c0a01edaa009b8962
Reviewed-on: http://gerrit.cloudera.org:8080/6422
Tested-by: Kudu Jenkins
Reviewed-by: David Ribeiro Alves <dralves@apache.org>
(cherry picked from commit 0450f77f69c74cc6dec08ad36bb89ac12c17608f)
Reviewed-on: http://gerrit.cloudera.org:8080/6572
Reviewed-by: Jean-Daniel Cryans <jdcryans@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/f6edcbac
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/f6edcbac
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/f6edcbac

Branch: refs/heads/branch-1.2.x
Commit: f6edcbac28c5cc7ca01f17832c201e75f34faea4
Parents: 5003991
Author: Mike Percy <mpercy@apache.org>
Authored: Fri Mar 17 15:19:36 2017 -0700
Committer: Jean-Daniel Cryans <jdcryans@apache.org>
Committed: Tue Apr 18 21:20:03 2017 +0000

----------------------------------------------------------------------
 src/kudu/integration-tests/CMakeLists.txt       |   1 +
 .../integration-tests/delete_tablet-itest.cc    | 106 +++++++++++++++++++
 .../integration-tests/external_mini_cluster.cc  |   6 +-
 .../integration-tests/external_mini_cluster.h   |  22 +---
 src/kudu/integration-tests/mini_cluster.cc      |  23 +++-
 src/kudu/integration-tests/mini_cluster.h       |   6 ++
 src/kudu/integration-tests/mini_cluster_base.h  |  22 +++-
 src/kudu/tablet/tablet_bootstrap.cc             |  78 +++++++++-----
 src/kudu/tserver/mini_tablet_server.cc          |   5 +
 src/kudu/tserver/mini_tablet_server.h           |   3 +
 10 files changed, 224 insertions(+), 48 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/CMakeLists.txt b/src/kudu/integration-tests/CMakeLists.txt
index fdea290..9e78795 100644
--- a/src/kudu/integration-tests/CMakeLists.txt
+++ b/src/kudu/integration-tests/CMakeLists.txt
@@ -54,6 +54,7 @@ ADD_KUDU_TEST(consistency-itest)
 ADD_KUDU_TEST(create-table-itest)
 ADD_KUDU_TEST(create-table-stress-test)
 ADD_KUDU_TEST(delete_table-test)
+ADD_KUDU_TEST(delete_tablet-itest)
 ADD_KUDU_TEST(disk_reservation-itest)
 ADD_KUDU_TEST(exactly_once_writes-itest)
 ADD_KUDU_TEST(external_mini_cluster-test RESOURCE_LOCK "master-rpc-ports")

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/delete_tablet-itest.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/delete_tablet-itest.cc b/src/kudu/integration-tests/delete_tablet-itest.cc
new file mode 100644
index 0000000..31fc1c7
--- /dev/null
+++ b/src/kudu/integration-tests/delete_tablet-itest.cc
@@ -0,0 +1,106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <vector>
+
+#include "kudu/gutil/stl_util.h"
+#include "kudu/gutil/strings/substitute.h"
+#include "kudu/integration-tests/cluster_itest_util.h"
+#include "kudu/integration-tests/mini_cluster-itest-base.h"
+#include "kudu/integration-tests/test_workload.h"
+#include "kudu/master/mini_master.h"
+#include "kudu/tablet/tablet_peer.h"
+#include "kudu/tserver/mini_tablet_server.h"
+#include "kudu/tserver/tablet_server.h"
+#include "kudu/tserver/ts_tablet_manager.h"
+
+DECLARE_int64(fs_wal_dir_reserved_bytes);
+
+using kudu::tablet::TabletPeer;
+using std::vector;
+
+namespace kudu {
+
+class DeleteTabletITest : public MiniClusterITestBase {
+};
+
+// Test deleting a failed replica. Regression test for KUDU-1607.
+TEST_F(DeleteTabletITest, TestDeleteFailedReplica) {
+  NO_FATALS(StartCluster(1)); // 1 TS.
+  TestWorkload workload(cluster_.get());
+  workload.set_num_replicas(1);
+  workload.Setup();
+
+  std::unordered_map<std::string, itest::TServerDetails*> ts_map;
+  ValueDeleter del(&ts_map);
+  ASSERT_OK(itest::CreateTabletServerMap(cluster_->master_proxy().get(),
+                                         cluster_->messenger(),
+                                         &ts_map));
+  auto* mts = cluster_->mini_tablet_server(0);
+  auto* ts = ts_map[mts->uuid()];
+
+  scoped_refptr<TabletPeer> tablet_peer;
+  AssertEventually([&] {
+    vector<scoped_refptr<TabletPeer>> tablet_peers;
+    mts->server()->tablet_manager()->GetTabletPeers(&tablet_peers);
+    ASSERT_EQ(1, tablet_peers.size());
+    tablet_peer = tablet_peers[0];
+  });
+  NO_FATALS();
+
+  workload.Start();
+  while (workload.rows_inserted() < 100) {
+    SleepFor(MonoDelta::FromMilliseconds(10));
+  }
+  workload.StopAndJoin();
+
+  // We need blocks on-disk for this regression test, so force an MRS flush.
+  ASSERT_OK(tablet_peer->tablet()->Flush());
+
+  // Shut down the master so it doesn't crash the test process when we make the
+  // disk appear to be full.
+  cluster_->mini_master()->Shutdown();
+
+  // Shut down the TS and restart it after changing flags to ensure no data can
+  // be written during tablet bootstrap.
+  mts->Shutdown();
+  FLAGS_fs_wal_dir_reserved_bytes = INT64_MAX;
+  ASSERT_OK(mts->Start());
+  Status s = mts->server()->tablet_manager()->WaitForAllBootstrapsToFinish();
+  ASSERT_TRUE(s.IsIOError());
+  ASSERT_STR_CONTAINS(s.ToString(), "Insufficient disk space");
+
+  // Tablet bootstrap failure should result in tablet status == FAILED.
+  {
+    vector<scoped_refptr<TabletPeer>> tablet_peers;
+    mts->server()->tablet_manager()->GetTabletPeers(&tablet_peers);
+    ASSERT_EQ(1, tablet_peers.size());
+    tablet_peer = tablet_peers[0];
+    ASSERT_EQ(tablet::FAILED, tablet_peer->state());
+  }
+
+  // We should still be able to delete the failed tablet.
+  ASSERT_OK(itest::DeleteTablet(ts, tablet_peer->tablet_id(), tablet::TABLET_DATA_DELETED,
+                                boost::none, MonoDelta::FromSeconds(30)));
+  AssertEventually([&] {
+    vector<scoped_refptr<TabletPeer>> tablet_peers;
+    mts->server()->tablet_manager()->GetTabletPeers(&tablet_peers);
+    ASSERT_EQ(0, tablet_peers.size());
+  });
+}
+
+} // namespace kudu

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/external_mini_cluster.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/external_mini_cluster.cc b/src/kudu/integration-tests/external_mini_cluster.cc
index 13c86d4..ed79e37 100644
--- a/src/kudu/integration-tests/external_mini_cluster.cc
+++ b/src/kudu/integration-tests/external_mini_cluster.cc
@@ -533,16 +533,16 @@ vector<ExternalDaemon*> ExternalMiniCluster::daemons() const {
   return results;
 }
 
-std::shared_ptr<rpc::Messenger> ExternalMiniCluster::messenger() {
+std::shared_ptr<rpc::Messenger> ExternalMiniCluster::messenger() const {
   return messenger_;
 }
 
-std::shared_ptr<MasterServiceProxy> ExternalMiniCluster::master_proxy() {
+std::shared_ptr<MasterServiceProxy> ExternalMiniCluster::master_proxy() const {
   CHECK_EQ(masters_.size(), 1);
   return master_proxy(0);
 }
 
-std::shared_ptr<MasterServiceProxy> ExternalMiniCluster::master_proxy(int idx) {
+std::shared_ptr<MasterServiceProxy> ExternalMiniCluster::master_proxy(int idx) const
{
   CHECK_LT(idx, masters_.size());
   return std::shared_ptr<MasterServiceProxy>(
       new MasterServiceProxy(messenger_, CHECK_NOTNULL(master(idx))->bound_rpc_addr()));

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/external_mini_cluster.h
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/external_mini_cluster.h b/src/kudu/integration-tests/external_mini_cluster.h
index 48979f0..5eb49f5 100644
--- a/src/kudu/integration-tests/external_mini_cluster.h
+++ b/src/kudu/integration-tests/external_mini_cluster.h
@@ -48,14 +48,6 @@ class NodeInstancePB;
 class Sockaddr;
 class Subprocess;
 
-namespace master {
-class MasterServiceProxy;
-} // namespace master
-
-namespace rpc {
-class Messenger;
-} // namespace rpc
-
 namespace server {
 class ServerStatusPB;
 } // namespace server
@@ -233,17 +225,9 @@ class ExternalMiniCluster : public MiniClusterBase {
     return masters_.size();
   }
 
-  // Return the client messenger used by the ExternalMiniCluster.
-  std::shared_ptr<rpc::Messenger> messenger();
-
-  // If the cluster is configured for a single non-distributed master,
-  // return a proxy to that master. Requires that the single master is
-  // running.
-  std::shared_ptr<master::MasterServiceProxy> master_proxy();
-
-  // Returns an RPC proxy to the master at 'idx'. Requires that the
-  // master at 'idx' is running.
-  std::shared_ptr<master::MasterServiceProxy> master_proxy(int idx);
+  std::shared_ptr<rpc::Messenger> messenger() const override;
+  std::shared_ptr<master::MasterServiceProxy> master_proxy() const override;
+  std::shared_ptr<master::MasterServiceProxy> master_proxy(int idx) const override;
 
   // Wait until the number of registered tablet servers reaches the given count
   // on all of the running masters. Returns Status::TimedOut if the desired

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/mini_cluster.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/mini_cluster.cc b/src/kudu/integration-tests/mini_cluster.cc
index d633151..30c5c57 100644
--- a/src/kudu/integration-tests/mini_cluster.cc
+++ b/src/kudu/integration-tests/mini_cluster.cc
@@ -23,6 +23,7 @@
 #include "kudu/gutil/strings/substitute.h"
 #include "kudu/master/catalog_manager.h"
 #include "kudu/master/master.h"
+#include "kudu/master/master.proxy.h"
 #include "kudu/master/mini_master.h"
 #include "kudu/master/ts_descriptor.h"
 #include "kudu/master/ts_manager.h"
@@ -41,8 +42,8 @@ namespace kudu {
 using client::KuduClient;
 using client::KuduClientBuilder;
 using master::CatalogManager;
+using master::MasterServiceProxy;
 using master::MiniMaster;
-using master::TabletLocationsPB;
 using master::TSDescriptor;
 using std::shared_ptr;
 using tserver::MiniTabletServer;
@@ -96,6 +97,12 @@ Status MiniCluster::Start() {
   RETURN_NOT_OK_PREPEND(WaitForTabletServerCount(num_ts_initial_),
                         "Waiting for tablet servers to start");
 
+  RETURN_NOT_OK_PREPEND(rpc::MessengerBuilder("minicluster-messenger")
+                        .set_num_reactors(1)
+                        .set_max_negotiation_threads(1)
+                        .Build(&messenger_),
+                        "Failed to start Messenger for minicluster");
+
   running_ = true;
   return Status::OK();
 }
@@ -322,4 +329,18 @@ Status MiniCluster::GetLeaderMasterIndex(int* idx) const {
   return Status::OK();
 }
 
+std::shared_ptr<rpc::Messenger> MiniCluster::messenger() const {
+  return messenger_;
+}
+
+std::shared_ptr<MasterServiceProxy> MiniCluster::master_proxy() const {
+  CHECK_EQ(1, mini_masters_.size());
+  return master_proxy(0);
+}
+
+std::shared_ptr<MasterServiceProxy> MiniCluster::master_proxy(int idx) const {
+  return std::shared_ptr<MasterServiceProxy>(
+      new MasterServiceProxy(messenger_, CHECK_NOTNULL(mini_master(idx))->bound_rpc_addr()));
+}
+
 } // namespace kudu

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/mini_cluster.h
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/mini_cluster.h b/src/kudu/integration-tests/mini_cluster.h
index a04a89c..65b87c3 100644
--- a/src/kudu/integration-tests/mini_cluster.h
+++ b/src/kudu/integration-tests/mini_cluster.h
@@ -156,6 +156,10 @@ class MiniCluster : public MiniClusterBase {
   // last result may not be valid.
   Status GetLeaderMasterIndex(int* idx) const;
 
+  std::shared_ptr<rpc::Messenger> messenger() const override;
+  std::shared_ptr<master::MasterServiceProxy> master_proxy() const override;
+  std::shared_ptr<master::MasterServiceProxy> master_proxy(int idx) const override;
+
  private:
   enum {
     kRegistrationWaitTimeSeconds = 15,
@@ -174,6 +178,8 @@ class MiniCluster : public MiniClusterBase {
   std::vector<std::shared_ptr<master::MiniMaster> > mini_masters_;
   std::vector<std::shared_ptr<tserver::MiniTabletServer> > mini_tablet_servers_;
 
+  std::shared_ptr<rpc::Messenger> messenger_;
+
   DISALLOW_COPY_AND_ASSIGN(MiniCluster);
 };
 

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/integration-tests/mini_cluster_base.h
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/mini_cluster_base.h b/src/kudu/integration-tests/mini_cluster_base.h
index 1fd1985..a71b807 100644
--- a/src/kudu/integration-tests/mini_cluster_base.h
+++ b/src/kudu/integration-tests/mini_cluster_base.h
@@ -24,7 +24,15 @@ class Status;
 namespace client {
 class KuduClient;
 class KuduClientBuilder;
-}
+} // namespace client
+
+namespace master {
+class MasterServiceProxy;
+} // namespace master
+
+namespace rpc {
+class Messenger;
+} // namespace rpc
 
 // Mode to which node types a certain action (like Shutdown()) should apply.
 enum class ClusterNodes {
@@ -65,6 +73,18 @@ class MiniClusterBase {
   // REQUIRES: the cluster must have already been Start()ed.
   virtual Status CreateClient(client::KuduClientBuilder* builder,
                               client::sp::shared_ptr<client::KuduClient>* client) const
= 0;
+
+  // Return a messenger for use by clients.
+  virtual std::shared_ptr<rpc::Messenger> messenger() const = 0;
+
+  // If the cluster is configured for a single non-distributed master,
+  // return a proxy to that master. Requires that the single master is
+  // running.
+  virtual std::shared_ptr<master::MasterServiceProxy> master_proxy() const = 0;
+
+  // Returns an RPC proxy to the master at 'idx'. Requires that the
+  // master at 'idx' is running.
+  virtual std::shared_ptr<master::MasterServiceProxy> master_proxy(int idx) const =
0;
 };
 
 } // namespace kudu

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/tablet/tablet_bootstrap.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/tablet_bootstrap.cc b/src/kudu/tablet/tablet_bootstrap.cc
index 8508e55..6a11ba5 100644
--- a/src/kudu/tablet/tablet_bootstrap.cc
+++ b/src/kudu/tablet/tablet_bootstrap.cc
@@ -57,6 +57,7 @@
 #include "kudu/util/logging.h"
 #include "kudu/util/path_util.h"
 #include "kudu/util/pb_util.h"
+#include "kudu/util/scoped_cleanup.h"
 #include "kudu/util/stopwatch.h"
 
 DEFINE_bool(skip_remove_old_recovery_dir, false,
@@ -176,10 +177,17 @@ class TabletBootstrap {
   // A successful call will yield the rebuilt tablet and the rebuilt log.
   Status Bootstrap(shared_ptr<Tablet>* rebuilt_tablet,
                    scoped_refptr<Log>* rebuilt_log,
-                   ConsensusBootstrapInfo* results);
+                   ConsensusBootstrapInfo* consensus_info);
 
  private:
 
+  // The method that does the actual work of tablet bootstrap. Bootstrap() is
+  // actually a wrapper method that is responsible for pinning and unpinning
+  // the tablet metadata flush.
+  Status RunBootstrap(shared_ptr<Tablet>* rebuilt_tablet,
+                      scoped_refptr<Log>* rebuilt_log,
+                      ConsensusBootstrapInfo* consensus_info);
+
   // Opens the tablet.
   // Sets '*has_blocks' to true if there was any data on disk for this tablet.
   Status OpenTablet(bool* has_blocks);
@@ -212,9 +220,9 @@ class TabletBootstrap {
   Status OpenNewLog();
 
   // Finishes bootstrap, setting 'rebuilt_log' and 'rebuilt_tablet'.
-  Status FinishBootstrap(const string& message,
-                         scoped_refptr<log::Log>* rebuilt_log,
-                         shared_ptr<Tablet>* rebuilt_tablet);
+  void FinishBootstrap(const string& message,
+                       scoped_refptr<log::Log>* rebuilt_log,
+                       shared_ptr<Tablet>* rebuilt_tablet);
 
   // Plays the log segments into the tablet being built.
   // The process of playing the segments generates a new log that can be continued
@@ -438,6 +446,43 @@ TabletBootstrap::TabletBootstrap(
 Status TabletBootstrap::Bootstrap(shared_ptr<Tablet>* rebuilt_tablet,
                                   scoped_refptr<Log>* rebuilt_log,
                                   ConsensusBootstrapInfo* consensus_info) {
+  // We pin (prevent) metadata flush at the beginning of the bootstrap process
+  // and always unpin it at the end.
+  meta_->PinFlush();
+
+  // Now run the actual bootstrap process.
+  Status bootstrap_status = RunBootstrap(rebuilt_tablet, rebuilt_log, consensus_info);
+
+  // Add a callback to TabletMetadata that makes sure that each time we flush the metadata
+  // we also wait for in-flights to finish and for their wal entry to be fsynced.
+  // This might be a bit conservative in some situations but it will prevent us from
+  // ever flushing the metadata referring to tablet data blocks containing data whose
+  // commit entries are not durable, a pre-requisite for recovery.
+  CHECK((*rebuilt_tablet && *rebuilt_log) || !bootstrap_status.ok())
+      << "Tablet and Log not initialized";
+  if (bootstrap_status.ok()) {
+    meta_->SetPreFlushCallback(
+        Bind(&FlushInflightsToLogCallback::WaitForInflightsAndFlushLog,
+             make_scoped_refptr(new FlushInflightsToLogCallback(
+                 rebuilt_tablet->get(), *rebuilt_log))));
+  }
+
+  // This will cause any pending TabletMetadata flush to be executed.
+  Status unpin_status = meta_->UnPinFlush();
+
+  constexpr char kFailedUnpinMsg[] = "Failed to flush after unpinning";
+  if (PREDICT_FALSE(!bootstrap_status.ok() && !unpin_status.ok())) {
+    LOG_WITH_PREFIX(WARNING) << kFailedUnpinMsg << ": " << unpin_status.ToString();
+    return bootstrap_status;
+  }
+  RETURN_NOT_OK(bootstrap_status);
+  RETURN_NOT_OK_PREPEND(unpin_status, Substitute("$0$1", LogPrefix(), kFailedUnpinMsg));
+  return Status::OK();
+}
+
+Status TabletBootstrap::RunBootstrap(shared_ptr<Tablet>* rebuilt_tablet,
+                                     scoped_refptr<Log>* rebuilt_log,
+                                     ConsensusBootstrapInfo* consensus_info) {
   string tablet_id = meta_->tablet_id();
 
   // Replay requires a valid Consensus metadata file to exist in order to
@@ -457,8 +502,6 @@ Status TabletBootstrap::Bootstrap(shared_ptr<Tablet>* rebuilt_tablet,
                               TabletDataState_Name(tablet_data_state));
   }
 
-  meta_->PinFlush();
-
   StatusMessage("Bootstrap starting.");
 
   if (VLOG_IS_ON(1)) {
@@ -482,9 +525,7 @@ Status TabletBootstrap::Bootstrap(shared_ptr<Tablet>* rebuilt_tablet,
   if (!has_blocks && !needs_recovery) {
     LOG_WITH_PREFIX(INFO) << "No blocks or log segments found. Creating new log.";
     RETURN_NOT_OK_PREPEND(OpenNewLog(), "Failed to open new log");
-    RETURN_NOT_OK(FinishBootstrap("No bootstrap required, opened a new log",
-                                  rebuilt_log,
-                                  rebuilt_tablet));
+    FinishBootstrap("No bootstrap required, opened a new log", rebuilt_log, rebuilt_tablet);
     consensus_info->last_id = MinimumOpId();
     consensus_info->last_committed_id = MinimumOpId();
     return Status::OK();
@@ -505,29 +546,18 @@ Status TabletBootstrap::Bootstrap(shared_ptr<Tablet>* rebuilt_tablet,
   cmeta_->Flush();
 
   RETURN_NOT_OK(RemoveRecoveryDir());
-  RETURN_NOT_OK(FinishBootstrap("Bootstrap complete.", rebuilt_log, rebuilt_tablet));
+  FinishBootstrap("Bootstrap complete.", rebuilt_log, rebuilt_tablet);
 
   return Status::OK();
 }
 
-Status TabletBootstrap::FinishBootstrap(const string& message,
-                                        scoped_refptr<log::Log>* rebuilt_log,
-                                        shared_ptr<Tablet>* rebuilt_tablet) {
-  // Add a callback to TabletMetadata that makes sure that each time we flush the metadata
-  // we also wait for in-flights to finish and for their wal entry to be fsynced.
-  // This might be a bit conservative in some situations but it will prevent us from
-  // ever flushing the metadata referring to tablet data blocks containing data whose
-  // commit entries are not durable, a pre-requisite for recovery.
-  meta_->SetPreFlushCallback(
-      Bind(&FlushInflightsToLogCallback::WaitForInflightsAndFlushLog,
-           make_scoped_refptr(new FlushInflightsToLogCallback(tablet_.get(),
-                                                              log_))));
+void TabletBootstrap::FinishBootstrap(const string& message,
+                                      scoped_refptr<log::Log>* rebuilt_log,
+                                      shared_ptr<Tablet>* rebuilt_tablet) {
   tablet_->MarkFinishedBootstrapping();
-  RETURN_NOT_OK(tablet_->metadata()->UnPinFlush());
   StatusMessage(message);
   rebuilt_tablet->reset(tablet_.release());
   rebuilt_log->swap(log_);
-  return Status::OK();
 }
 
 Status TabletBootstrap::OpenTablet(bool* has_blocks) {

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/tserver/mini_tablet_server.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tserver/mini_tablet_server.cc b/src/kudu/tserver/mini_tablet_server.cc
index 92f1ad2..ee08332 100644
--- a/src/kudu/tserver/mini_tablet_server.cc
+++ b/src/kudu/tserver/mini_tablet_server.cc
@@ -151,5 +151,10 @@ const Sockaddr MiniTabletServer::bound_http_addr() const {
   return server_->first_http_address();
 }
 
+const string& MiniTabletServer::uuid() const {
+  CHECK(started_);
+  return server_->fs_manager()->uuid();
+}
+
 } // namespace tserver
 } // namespace kudu

http://git-wip-us.apache.org/repos/asf/kudu/blob/f6edcbac/src/kudu/tserver/mini_tablet_server.h
----------------------------------------------------------------------
diff --git a/src/kudu/tserver/mini_tablet_server.h b/src/kudu/tserver/mini_tablet_server.h
index 4c205fd..34ed945 100644
--- a/src/kudu/tserver/mini_tablet_server.h
+++ b/src/kudu/tserver/mini_tablet_server.h
@@ -87,6 +87,9 @@ class MiniTabletServer {
   const TabletServer* server() const { return server_.get(); }
   TabletServer* server() { return server_.get(); }
 
+  // Return TS uuid.
+  const std::string& uuid() const;
+
   bool is_started() const { return started_; }
 
   void FailHeartbeats();


Mime
View raw message