Hi Aaron,
Thanks for your answer.
I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes
instead of 16 (it is a test cluster).
There are 2 datacenters b1 and s1.
Here is the result of nodetool status after adding a new node in the 1st
datacenter (dc s1):
root@node007:~# nodetool status
Datacenter: b1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.234.72.135 10.71 GB 256 44.6%
2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
UN 10.234.72.134 16.74 GB 256 63.7%
f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
UN 10.234.72.139 17.09 GB 256 62.0%
95661392-ccd8-4592-a76f-1c99f7cdf23a e07
UN 10.234.72.138 10.96 GB 256 42.9%
0d6725f0-1357-423d-85c1-153fb94257d5 e03
UN 10.234.72.137 11.09 GB 256 45.7%
492190d7-3055-4167-8699-9c6560e28164 e03
UN 10.234.72.136 11.91 GB 256 41.1%
3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
Datacenter: s1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.98.255.139 16.94 GB 256 43.8%
3523e80c-8468-4502-b334-79eabc3357f0 g10
UN 10.98.255.138 12.62 GB 256 42.4%
a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
UN 10.98.255.137 10.59 GB 256 38.4%
f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
UN 10.98.255.136 11.89 GB 256 42.9%
36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
UN 10.98.255.135 10.29 GB 256 40.4%
e2d020a5-97a9-48d4-870c-d10b59858763 a09
UN 10.98.255.134 16.19 GB 256 52.3%
73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
UN 10.98.255.140 127.84 KB 256 39.9%
3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
We can see that the new node (10.98.255.140) contains only 127,84KB.
We saw also that there was no network traffic between the nodes.
Then we added a new node in the 2nd datacenter (dc b1)
root@node007:~# nodetool status
Datacenter: b1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.234.72.135 12.95 GB 256 42.0%
2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
UN 10.234.72.134 20.11 GB 256 53.1%
f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
UN 10.234.72.140 122.25 KB 256 41.9%
501ea498-8fed-4cc8-a23a-c99492bc4f26 e07
UN 10.234.72.139 20.46 GB 256 40.2%
95661392-ccd8-4592-a76f-1c99f7cdf23a e07
UN 10.234.72.138 13.21 GB 256 40.9%
0d6725f0-1357-423d-85c1-153fb94257d5 e03
UN 10.234.72.137 13.34 GB 256 42.9%
492190d7-3055-4167-8699-9c6560e28164 e03
UN 10.234.72.136 14.16 GB 256 39.0%
3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
Datacenter: s1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.98.255.139 19.19 GB 256 43.8%
3523e80c-8468-4502-b334-79eabc3357f0 g10
UN 10.98.255.138 14.9 GB 256 42.4%
a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
UN 10.98.255.137 12.49 GB 256 38.4%
f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
UN 10.98.255.136 14.13 GB 256 42.9%
36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
UN 10.98.255.135 12.16 GB 256 40.4%
e2d020a5-97a9-48d4-870c-d10b59858763 a09
UN 10.98.255.134 18.85 GB 256 52.3%
73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
UN 10.98.255.140 2.24 GB 256 39.9%
3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
The new node in the 1st datacenter contains now 2,24 GB because we were
inserting data in the cluster while adding the new nodes.
Then we started a repair from the new node in the 2nd datacenter :
time nodetool repair
We can see that the old nodes are sending data to the new node :
root@node007:~# nodetool netstats
Mode: NORMAL
Not sending any streams.
Streaming from: /10.98.255.137
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
sections=130 progress=0/15598366 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
sections=107 progress=0/429517 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=109 progress=0/696057 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
sections=57 progress=0/189844 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
sections=124 progress=56492032/4597955 - 1228%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
sections=113 progress=0/850632 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
sections=130 progress=0/16815423 - 0%
Streaming from: /10.98.255.136
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=104 progress=0/3545520 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=43 progress=0/252976 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=91 progress=0/443365 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
sections=101 progress=11337728/4188624 - 270%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
sections=94 progress=0/758418 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
sections=88 progress=0/396552 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
sections=105 progress=0/12363907 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=87 progress=0/506537 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
sections=42 progress=0/197531 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
sections=110 progress=0/12689898 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=83 progress=0/379628 - 0%
Streaming from: /10.98.255.134
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=108 progress=0/10715348 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=82 progress=0/8177161 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=84 progress=25613959168/8556942 - 299335%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=79 progress=0/506293 - 0%
Pool Name Active Pending Completed
Commands n/a 0 835
Responses n/a 0 6765
So, it looks that the repair is required if we want to add new nodes in our
platform, but I don't understand why.
Regards.
Jean Armel
|