kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject [3/4] incubator-kudu git commit: sys_catalog: another band-aid to avoid races while shutting down
Date Tue, 03 May 2016 00:17:40 GMT
sys_catalog: another band-aid to avoid races while shutting down

I observed a test failure where the CHECK in SysCatalogStateChanged fired
because a master leader election finished just as the master was shut down.
To fix, we'll soften the CHECK to allow (and ignore) failures that happen
after the master has shut down.

Check failed: _s.ok() Bad status: Service unavailable: The pool has been shut down.
*** Check failure stack trace: ***
    @     0x7f73820ad24d  google::LogMessage::Fail() at ??:0
    @     0x7f73820af2bc  google::LogMessage::SendToLog() at ??:0
    @     0x7f73820acda9  google::LogMessage::Flush() at ??:0
    @     0x7f73820afc4f  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f738f0c9220  kudu::master::SysCatalogTable::SysCatalogStateChanged() at ??:0
    @     0x7f738f0d98fb  kudu::internal::InvokeHelper<>::MakeItSo() at ??:0
    @     0x7f738f0d9616  kudu::internal::Invoker<>::Run() at ??:0
    @     0x7f738cb50148  kudu::internal::Invoker<>::Run() at ??:0
    @     0x7f738cb0442e  boost::_bi::bind_t<>::operator()() at ??:0
    @     0x7f738659db6f  boost::function0<>::operator()() at ??:0
    @     0x7f7383657d3c  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f738365f3ee  boost::_bi::bind_t<>::operator()() at ??:0
    @     0x7f738659db6f  boost::function0<>::operator()() at ??:0
    @     0x7f73836456b7  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f738a591182  start_thread at ??:0
    @     0x7f73808e547d  clone at ??:0
    @              (nil)  (unknown)

Change-Id: Iaf6cddee882bbaf3cf9b7198926799d013e14253
Reviewed-on: http://gerrit.cloudera.org:8080/2928
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <todd@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/40900879
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/40900879
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/40900879

Branch: refs/heads/master
Commit: 40900879fd7ff4fd49ec59ffa48558ae22639a92
Parents: 1681d9a
Author: Adar Dembo <adar@cloudera.com>
Authored: Mon May 2 21:58:32 2016 +0200
Committer: Todd Lipcon <todd@apache.org>
Committed: Tue May 3 00:17:15 2016 +0000

----------------------------------------------------------------------
 src/kudu/master/sys_catalog.cc | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/40900879/src/kudu/master/sys_catalog.cc
----------------------------------------------------------------------
diff --git a/src/kudu/master/sys_catalog.cc b/src/kudu/master/sys_catalog.cc
index 9e5b9b6..a9daa96 100644
--- a/src/kudu/master/sys_catalog.cc
+++ b/src/kudu/master/sys_catalog.cc
@@ -240,7 +240,12 @@ void SysCatalogTable::SysCatalogStateChanged(const string& tablet_id,
const stri
                         << RaftPeerPB::Role_Name(new_role)
                         << ", previous role was: " << RaftPeerPB::Role_Name(old_role_);
   if (new_role == RaftPeerPB::LEADER) {
-    CHECK_OK(leader_cb_.Run());
+    Status s = leader_cb_.Run();
+
+    // Callback errors are non-fatal only if the catalog manager has shut down.
+    if (!s.ok()) {
+      CHECK(!master_->catalog_manager()->IsInitialized()) << s.ToString();
+    }
   }
 }
 


Mime
View raw message