incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Armel Luce <jaluc...@gmail.com>
Subject Re: Adding new nodes in a cluster with virtual nodes
Date Fri, 22 Feb 2013 08:55:17 GMT
Hi Aaron,

Thanks for your answer.


I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes
instead of 16 (it is a test cluster).
There are 2 datacenters b1 and s1.

Here is the result of nodetool status after adding a new node in the 1st
datacenter (dc s1):
root@node007:~# nodetool status
Datacenter: b1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host
ID                               Rack
UN  10.234.72.135     10.71 GB   256     44.6%
2fc583b2-822f-4347-9fab-5e9d10d548c9  c01
UN  10.234.72.134     16.74 GB   256     63.7%
f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1  e01
UN  10.234.72.139     17.09 GB   256     62.0%
95661392-ccd8-4592-a76f-1c99f7cdf23a  e07
UN  10.234.72.138     10.96 GB   256     42.9%
0d6725f0-1357-423d-85c1-153fb94257d5  e03
UN  10.234.72.137     11.09 GB   256     45.7%
492190d7-3055-4167-8699-9c6560e28164  e03
UN  10.234.72.136     11.91 GB   256     41.1%
3872f26c-5f2d-4fb3-9f5c-08b4c7762466  c01
Datacenter: s1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host
ID                               Rack
UN  10.98.255.139     16.94 GB   256     43.8%
3523e80c-8468-4502-b334-79eabc3357f0  g10
UN  10.98.255.138     12.62 GB   256     42.4%
a2bcddf1-393e-453b-9d4f-9f7111c01d7f  i02
UN  10.98.255.137     10.59 GB   256     38.4%
f851b6ee-f1e4-431b-8beb-e7b173a77342  i02
UN  10.98.255.136     11.89 GB   256     42.9%
36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e  a09
UN  10.98.255.135     10.29 GB   256     40.4%
e2d020a5-97a9-48d4-870c-d10b59858763  a09
UN  10.98.255.134     16.19 GB   256     52.3%
73e3376a-5a9f-4b8a-a119-c87ae1fafdcb  h06
UN  10.98.255.140     127.84 KB  256     39.9%
3d5c33e6-35d0-40a0-b60d-2696fd5cbf72  g10

We can see that the new node (10.98.255.140) contains only 127,84KB.
We saw also that there was no network traffic between the nodes.

Then we added a new node in the 2nd datacenter (dc b1)



root@node007:~# nodetool status
Datacenter: b1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host
ID                               Rack
UN  10.234.72.135     12.95 GB   256     42.0%
2fc583b2-822f-4347-9fab-5e9d10d548c9  c01
UN  10.234.72.134     20.11 GB   256     53.1%
f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1  e01
UN  10.234.72.140     122.25 KB  256     41.9%
501ea498-8fed-4cc8-a23a-c99492bc4f26  e07
UN  10.234.72.139     20.46 GB   256     40.2%
95661392-ccd8-4592-a76f-1c99f7cdf23a  e07
UN  10.234.72.138     13.21 GB   256     40.9%
0d6725f0-1357-423d-85c1-153fb94257d5  e03
UN  10.234.72.137     13.34 GB   256     42.9%
492190d7-3055-4167-8699-9c6560e28164  e03
UN  10.234.72.136     14.16 GB   256     39.0%
3872f26c-5f2d-4fb3-9f5c-08b4c7762466  c01
Datacenter: s1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host
ID                               Rack
UN  10.98.255.139     19.19 GB   256     43.8%
3523e80c-8468-4502-b334-79eabc3357f0  g10
UN  10.98.255.138     14.9 GB    256     42.4%
a2bcddf1-393e-453b-9d4f-9f7111c01d7f  i02
UN  10.98.255.137     12.49 GB   256     38.4%
f851b6ee-f1e4-431b-8beb-e7b173a77342  i02
UN  10.98.255.136     14.13 GB   256     42.9%
36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e  a09
UN  10.98.255.135     12.16 GB   256     40.4%
e2d020a5-97a9-48d4-870c-d10b59858763  a09
UN  10.98.255.134     18.85 GB   256     52.3%
73e3376a-5a9f-4b8a-a119-c87ae1fafdcb  h06
UN  10.98.255.140     2.24 GB    256     39.9%
3d5c33e6-35d0-40a0-b60d-2696fd5cbf72  g10


We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
The new node in the 1st datacenter contains now 2,24 GB because we were
inserting data in the cluster while adding the new nodes.

Then we started a repair from the new node in the 2nd datacenter :
time nodetool repair


We can see that the old nodes are sending data to the new node :
root@node007:~# nodetool netstats
Mode: NORMAL
Not sending any streams.
Streaming from: /10.98.255.137
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
sections=130 progress=0/15598366 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
sections=107 progress=0/429517 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=109 progress=0/696057 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
sections=57 progress=0/189844 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
sections=124 progress=56492032/4597955 - 1228%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
sections=113 progress=0/850632 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
sections=130 progress=0/16815423 - 0%
Streaming from: /10.98.255.136
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=104 progress=0/3545520 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=43 progress=0/252976 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=91 progress=0/443365 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
sections=101 progress=11337728/4188624 - 270%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
sections=94 progress=0/758418 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
sections=88 progress=0/396552 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
sections=105 progress=0/12363907 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=87 progress=0/506537 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
sections=42 progress=0/197531 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
sections=110 progress=0/12689898 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=83 progress=0/379628 - 0%
Streaming from: /10.98.255.134
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=108 progress=0/10715348 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=82 progress=0/8177161 - 0%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=84 progress=25613959168/8556942 - 299335%
   hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=79 progress=0/506293 - 0%
Pool Name                    Active   Pending      Completed
Commands                        n/a         0            835
Responses                       n/a         0           6765

So, it looks that the repair is required if we want to add new nodes in our
platform, but I don't understand why.

Regards.

Jean Armel

Mime
View raw message