kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a...@apache.org
Subject [1/2] kudu git commit: KUDU-2170 Masters can start with duplicates specified
Date Fri, 20 Oct 2017 00:40:53 GMT
Repository: kudu
Updated Branches:
  refs/heads/master f2eb6c277 -> 4172983fd


KUDU-2170 Masters can start with duplicates specified

Users can specify duplicate masters in --master_addresses, where a
duplicate includes duplicated names and the same server known by
different names. This doesn't cause problems as far as I know, but it
looks scary because, for example, the web ui shows extra masters listed in
/masters. This is because /masters shows one master per result from
ListMasters. This patch deduplicates the output of ListMasters by UUID.

It also warns on start if there are exact string duplicates in
--master_addresses, at the time the master compares on-disk and
configured master addresses, but before the masters are ready to
resolve UUIDs.

Also, the error message when on-disk and configured masters don't match
is confusing. This patch makes it easy to understand.

Change-Id: I5047ccf92dde5377da18a1a84e105a951fa5b48c
Reviewed-on: http://gerrit.cloudera.org:8080/8341
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/c7fa4aaf
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/c7fa4aaf
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/c7fa4aaf

Branch: refs/heads/master
Commit: c7fa4aaf704e4035843bc8fe20e5ecada2ed1acc
Parents: f2eb6c2
Author: Will Berkeley <wdberkeley@apache.org>
Authored: Thu Oct 19 13:34:02 2017 -0700
Committer: Will Berkeley <wdberkeley@gmail.com>
Committed: Fri Oct 20 00:37:55 2017 +0000

----------------------------------------------------------------------
 src/kudu/master/master.cc      | 14 +++++++++++++-
 src/kudu/master/sys_catalog.cc | 13 +++++++++++--
 2 files changed, 24 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/c7fa4aaf/src/kudu/master/master.cc
----------------------------------------------------------------------
diff --git a/src/kudu/master/master.cc b/src/kudu/master/master.cc
index 4a88007..0b2d7d9 100644
--- a/src/kudu/master/master.cc
+++ b/src/kudu/master/master.cc
@@ -18,8 +18,11 @@
 #include "kudu/master/master.h"
 
 #include <algorithm>
+#include <iterator>
 #include <memory>
 #include <ostream>
+#include <set>
+#include <string>
 #include <type_traits>
 #include <vector>
 
@@ -28,10 +31,12 @@
 
 #include "kudu/cfile/block_cache.h"
 #include "kudu/common/wire_protocol.h"
+#include "kudu/common/wire_protocol.pb.h"
 #include "kudu/consensus/metadata.pb.h"
 #include "kudu/fs/fs_manager.h"
 #include "kudu/gutil/bind.h"
 #include "kudu/gutil/bind_helpers.h"
+#include "kudu/gutil/map-util.h"
 #include "kudu/gutil/move.h"
 #include "kudu/gutil/strings/substitute.h"
 #include "kudu/master/catalog_manager.h"
@@ -303,6 +308,12 @@ Status Master::ListMasters(std::vector<ServerEntryPB>* masters)
const {
     return Status::OK();
   }
 
+  // Since --master_addresses may contain duplicates, including different names
+  // for the same server, we deduplicate the masters by UUID here.
+  auto uuid_cmp = [](const ServerEntryPB& left, const ServerEntryPB& right) {
+    return left.instance_id().permanent_uuid() < right.instance_id().permanent_uuid();
+  };
+  std::set<ServerEntryPB, decltype(uuid_cmp)> masters_by_uuid(uuid_cmp);
   for (const HostPort& peer_addr : opts_.master_addresses) {
     ServerEntryPB peer_entry;
     Status s = GetMasterEntryForHost(messenger_, peer_addr, &peer_entry);
@@ -313,9 +324,10 @@ Status Master::ListMasters(std::vector<ServerEntryPB>* masters)
const {
       LOG(WARNING) << s.ToString();
       StatusToPB(s, peer_entry.mutable_error());
     }
-    masters->push_back(peer_entry);
+    InsertIfNotPresent(&masters_by_uuid, peer_entry);
   }
 
+  std::copy(masters_by_uuid.begin(), masters_by_uuid.end(), std::back_inserter(*masters));
   return Status::OK();
 }
 

http://git-wip-us.apache.org/repos/asf/kudu/blob/c7fa4aaf/src/kudu/master/sys_catalog.cc
----------------------------------------------------------------------
diff --git a/src/kudu/master/sys_catalog.cc b/src/kudu/master/sys_catalog.cc
index fe16c96..7836380 100644
--- a/src/kudu/master/sys_catalog.cc
+++ b/src/kudu/master/sys_catalog.cc
@@ -24,6 +24,7 @@
 #include <memory>
 #include <ostream>
 #include <set>
+#include <string>
 #include <type_traits>
 #include <utility>
 #include <vector>
@@ -196,6 +197,11 @@ Status SysCatalogTable::Load(FsManager *fs_manager) {
     for (const auto& hp : master_->opts().master_addresses) {
       peer_addrs_from_opts.insert(hp.ToString());
     }
+    if (peer_addrs_from_opts.size() < master_->opts().master_addresses.size()) {
+      LOG(WARNING) << Substitute("Found duplicates in --master_addresses: "
+                                 "the unique set of addresses is $0",
+                                 JoinStrings(peer_addrs_from_opts, ", "));
+    }
     set<string> peer_addrs_from_disk;
     for (const auto& p : cstate.committed_config().peers()) {
       HostPort hp;
@@ -210,8 +216,11 @@ Status SysCatalogTable::Load(FsManager *fs_manager) {
                                   std::back_inserter(symm_diff));
     if (!symm_diff.empty()) {
       string msg = Substitute(
-          "on-disk and provided master lists are different: $0",
-          JoinStrings(symm_diff, " "));
+          "on-disk master list ($0) and provided master list ($1) differ. "
+          "Their symmetric difference is: $2",
+          JoinStrings(peer_addrs_from_opts, ", "),
+          JoinStrings(peer_addrs_from_disk, ", "),
+          JoinStrings(symm_diff, ", "));
       return Status::InvalidArgument(msg);
     }
   }


Mime
View raw message