incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Adding new nodes in a cluster with virtual nodes
Date Fri, 22 Feb 2013 17:05:30 GMT
> So, it looks that the repair is required if we want to add new nodes in our platform,
but I don't understand why.
Bootstrapping should take care of it. But new seed nodes do not bootstrap. Check the logs
on the nodes you added to see what messages have "bootstrap" in them. 

Anytime you are worried about things like this throw in a nodetool repair. If you are using
QUOURM for read and writes you will still be getting consistent data, so long as you have
only added one node. Or one node every RF'th nodes. 

Cheers


-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 22/02/2013, at 9:55 PM, Jean-Armel Luce <jaluce06@gmail.com> wrote:

> Hi Aaron,
> 
> Thanks for your answer.
> 
> 
> I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes instead of
16 (it is a test cluster).
> There are 2 datacenters b1 and s1.
> 
> Here is the result of nodetool status after adding a new node in the 1st datacenter (dc
s1):
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  10.234.72.135     10.71 GB   256     44.6%             2fc583b2-822f-4347-9fab-5e9d10d548c9
 c01
> UN  10.234.72.134     16.74 GB   256     63.7%             f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1
 e01
> UN  10.234.72.139     17.09 GB   256     62.0%             95661392-ccd8-4592-a76f-1c99f7cdf23a
 e07
> UN  10.234.72.138     10.96 GB   256     42.9%             0d6725f0-1357-423d-85c1-153fb94257d5
 e03
> UN  10.234.72.137     11.09 GB   256     45.7%             492190d7-3055-4167-8699-9c6560e28164
 e03
> UN  10.234.72.136     11.91 GB   256     41.1%             3872f26c-5f2d-4fb3-9f5c-08b4c7762466
 c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  10.98.255.139     16.94 GB   256     43.8%             3523e80c-8468-4502-b334-79eabc3357f0
 g10
> UN  10.98.255.138     12.62 GB   256     42.4%             a2bcddf1-393e-453b-9d4f-9f7111c01d7f
 i02
> UN  10.98.255.137     10.59 GB   256     38.4%             f851b6ee-f1e4-431b-8beb-e7b173a77342
 i02
> UN  10.98.255.136     11.89 GB   256     42.9%             36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e
 a09
> UN  10.98.255.135     10.29 GB   256     40.4%             e2d020a5-97a9-48d4-870c-d10b59858763
 a09
> UN  10.98.255.134     16.19 GB   256     52.3%             73e3376a-5a9f-4b8a-a119-c87ae1fafdcb
 h06
> UN  10.98.255.140     127.84 KB  256     39.9%             3d5c33e6-35d0-40a0-b60d-2696fd5cbf72
 g10
> 
> We can see that the new node (10.98.255.140) contains only 127,84KB.
> We saw also that there was no network traffic between the nodes.
> 
> Then we added a new node in the 2nd datacenter (dc b1)
> 
> 
> 
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  10.234.72.135     12.95 GB   256     42.0%             2fc583b2-822f-4347-9fab-5e9d10d548c9
 c01
> UN  10.234.72.134     20.11 GB   256     53.1%             f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1
 e01
> UN  10.234.72.140     122.25 KB  256     41.9%             501ea498-8fed-4cc8-a23a-c99492bc4f26
 e07
> UN  10.234.72.139     20.46 GB   256     40.2%             95661392-ccd8-4592-a76f-1c99f7cdf23a
 e07
> UN  10.234.72.138     13.21 GB   256     40.9%             0d6725f0-1357-423d-85c1-153fb94257d5
 e03
> UN  10.234.72.137     13.34 GB   256     42.9%             492190d7-3055-4167-8699-9c6560e28164
 e03
> UN  10.234.72.136     14.16 GB   256     39.0%             3872f26c-5f2d-4fb3-9f5c-08b4c7762466
 c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  10.98.255.139     19.19 GB   256     43.8%             3523e80c-8468-4502-b334-79eabc3357f0
 g10
> UN  10.98.255.138     14.9 GB    256     42.4%             a2bcddf1-393e-453b-9d4f-9f7111c01d7f
 i02
> UN  10.98.255.137     12.49 GB   256     38.4%             f851b6ee-f1e4-431b-8beb-e7b173a77342
 i02
> UN  10.98.255.136     14.13 GB   256     42.9%             36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e
 a09
> UN  10.98.255.135     12.16 GB   256     40.4%             e2d020a5-97a9-48d4-870c-d10b59858763
 a09
> UN  10.98.255.134     18.85 GB   256     52.3%             73e3376a-5a9f-4b8a-a119-c87ae1fafdcb
 h06
> UN  10.98.255.140     2.24 GB    256     39.9%             3d5c33e6-35d0-40a0-b60d-2696fd5cbf72
 g10
> 
> 
> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
> The new node in the 1st datacenter contains now 2,24 GB because we were inserting data
in the cluster while adding the new nodes.
> 
> Then we started a repair from the new node in the 2nd datacenter :
> time nodetool repair
> 
> 
> We can see that the old nodes are sending data to the new node :
> root@node007:~# nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Streaming from: /10.98.255.137
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
sections=130 progress=0/15598366 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
sections=107 progress=0/429517 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=109 progress=0/696057 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
sections=57 progress=0/189844 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
sections=124 progress=56492032/4597955 - 1228%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
sections=113 progress=0/850632 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
sections=130 progress=0/16815423 - 0%
> Streaming from: /10.98.255.136
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=104 progress=0/3545520 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=43 progress=0/252976 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=91 progress=0/443365 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
sections=101 progress=11337728/4188624 - 270%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
sections=94 progress=0/758418 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
sections=88 progress=0/396552 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
sections=105 progress=0/12363907 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=87 progress=0/506537 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
sections=42 progress=0/197531 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
sections=110 progress=0/12689898 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=83 progress=0/379628 - 0%
> Streaming from: /10.98.255.134
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=108 progress=0/10715348 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=82 progress=0/8177161 - 0%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=84 progress=25613959168/8556942 - 299335%
>    hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=79 progress=0/506293 - 0%
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0            835
> Responses                       n/a         0           6765
> 
> So, it looks that the repair is required if we want to add new nodes in our platform,
but I don't understand why.
> 
> Regards.
> 
> Jean Armel
> 
> 


Mime
View raw message