kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergejs Andrejevs <S.Andrej...@intrum.com>
Subject Adding a new kudu master
Date Mon, 02 Jul 2018 11:36:11 GMT
Hi there,

I am very glad Kudu is evolving so rapidly. Thanks for your contributions!

I have faced with a challenge:
I need to upgrade (reinstall) prod servers, where 3 Kudu masters are running. What would be
the best way to do it from Kudu perspective?
If it is not officially supported yet, could you advise a way, which minimizes the risks?

Environment/conditions:
Cloudera 5.14
Kudu 1.6
High-level procedure: remove 1 server from cluster, upgrade, return back to CM cluster, check,
proceed with the next server.
Some downtime is possible (let's say < 1h)

Approach:

I have already tried out at test cluster the steps, which were used to migrate from a single-master
to multi-master cluster (see the plan below). However, there was a remark not to use it in
order to add new nodes for 3+ master cluster.
Therefore, what could be an alternative way? If no alternatives, what could be the extra steps
to pay additional attention to check the status if Kudu cluster is in a good shape?
Any comments/suggestions are extremely appreciated as well.

Current plan:

0.       Cluster check

1.       Stop all masters (let's call them master-1, master-2, master-3).

2.       Remove from CM one Kudu master, e.g. master-3.

3.       Update raft meta by removing "master-3" from Kudu cluster (to be able to restart
Kudu):
sudo -u kudu kudu local_replica cmeta rewrite_raft_config 00000000000000000000000000000000
1234567890:master-1:7051 0987654321:master-2:7051
By the way, do I understand right that tablet_id 00000000000000000000000000000000 is a special,
containing cluster meta info?

4.       Start all masters. From now Kudu temporary consists of 2 masters.

5.       Cluster check.

6.       Upgrade the excluded server

7.       Stop all masters.

8.       Prepare "master-3" as Kudu master:
sudo -u kudu kudu fs format --fs_wal_dir=...  --fs_data_dirs=...
sudo -u kudu kudu fs dump uuid --fs_wal_dir=...  --fs_data_dirs=... 2>/dev/null
Let's say obtained id is 7777777777.
Add master-3 to CM.

9.       Run metainfo update at existing masters, i.e. master-1 and master-2:

sudo -u kudu kudu local_replica cmeta rewrite_raft_config 00000000000000000000000000000000
1234567890:master-1:7051 0987654321:master-2:7051 7777777777:master-3:7051

10.   Start one master, e.g. master-1.

Copy the current cluster state from master-1 to master-3:
sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=...  --fs_data_dirs=... 00000000000000000000000000000000
1234567890:master-1:7051

11.   Start remaining Kudu masters: master-2 and master-3.

12.   Cluster check.

* Optionally, at first there may be added 1 extra node (to increase from 3 to 4 the initial
number of Kudu masters, so that after removal of 1 node there are still HA with quorum of
3 masters). In this case steps 7-12 should be repeated and additionally HiveMetaStore update
should be executed:
UPDATE hive_meta_store_database.TABLE_PARAMS
SET PARAM_VALUE = 'master-1,master-2,master-3'
WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'master-1,master-2,master-3,master-4';
              After upgrades, the master-4 node to be removed by running steps 1-5.

Thanks!

Best regards,
Sergejs Andrejevs
Information about how we process personal data<http://www.intrum.com/privacy>


Mime
View raw message