cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcel Villet <marc...@ctlab.com>
Subject "nodetool getendpoints" is inconsistent when cluster restarts
Date Fri, 16 Feb 2018 12:19:52 GMT
Hi All

I saw something curious when I restarted our cluster following a full shutdown whereby the
result of calling nodetool getendpoints showed inconsistent results as each node was started
in turn.

Prior to shutdown, we had the following:

$ nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                         
     Rack
UN  10.0.3.11  46.53 GiB  256          51.1%             35dcbc33-efb3-4c98-9063-aa451e6e0c7f
 rack1
UN  10.0.3.12  44.44 GiB  256          48.9%             8c13e94a-2c8e-43a3-9fb0-11d4b5b4606f
 rack1
UN  10.0.3.21  41.42 GiB  256          45.4%             6b619cc8-e398-4e48-8542-09d84c3710f6
 rack2
UN  10.0.3.22  49.55 GiB  256          54.6%             962b04ee-e135-4ed5-87f9-7463c0b18860
 rack2
UN  10.0.3.31  45.76 GiB  256          50.2%             2a90ac41-a741-472f-8c74-a382a24eb65f
 rack3
UN  10.0.3.32  45.2 GiB   256          49.8%             e7dc5e68-bbde-4f6b-9426-43cf13c679c8
 rack3

$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.12
10.0.3.22
10.0.3.32

As each node was started up, the results differed as follows:

administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11

administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
10.0.3.12

administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
10.0.3.12
10.0.3.21

administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
10.0.3.12
10.0.3.22

administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.12
10.0.3.22
10.0.3.31

administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.12
10.0.3.22
10.0.3.32

Eventually, the endpoints settles on the expected values.

Seeing as the token ranges are stored in system.peers (stored on each node), why would I see
different endpoints as I do?

Regards
Marcel Villet
Osprey Pro Team Leader and Lead Architect
CT LAB (PTY) LTD - Bringing Quality To Power
T  +27 (0)21-880-9915  |   F  +27 (0)21-880-1088  |  www.ctlab.com<http://www.ctlab.com/>
 |  www.pq-portal.com<http://www.pq-portal.com/rdPage.aspx>
15 Termo Lane, Techno Park, Stellenbosch, South Africa, 7600  |  PO Box 897, Stellenbosch,7599.
[cid:E9CDFBB1-258E-48CA-8E20-3ED934168E78]





This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they are addressed. If you have received this email
in error please notify the system manager:administrator@ctlab.com

Mime
View raw message