incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Armel Luce <jaluc...@gmail.com>
Subject Re: Adding new nodes in a cluster with virtual nodes
Date Fri, 22 Feb 2013 17:32:59 GMT
Thanks Aaron.

I shall investigate more next week about this.

regards.

Jean Armel

2013/2/22 aaron morton <aaron@thelastpickle.com>

> So, it looks that the repair is required if we want to add new nodes in
> our platform, but I don't understand why.
>
> Bootstrapping should take care of it. But new seed nodes do not bootstrap.
> Check the logs on the nodes you added to see what messages have "bootstrap"
> in them.
>
> Anytime you are worried about things like this throw in a nodetool repair.
> If you are using QUOURM for read and writes you will still be getting
> consistent data, so long as you have only added one node. Or one node every
> RF'th nodes.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/02/2013, at 9:55 PM, Jean-Armel Luce <jaluce06@gmail.com> wrote:
>
> Hi Aaron,
>
> Thanks for your answer.
>
>
> I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes
> instead of 16 (it is a test cluster).
> There are 2 datacenters b1 and s1.
>
> Here is the result of nodetool status after adding a new node in the 1st
> datacenter (dc s1):
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host
> ID                               Rack
> UN  10.234.72.135     10.71 GB   256     44.6%
> 2fc583b2-822f-4347-9fab-5e9d10d548c9  c01
> UN  10.234.72.134     16.74 GB   256     63.7%
> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1  e01
> UN  10.234.72.139     17.09 GB   256     62.0%
> 95661392-ccd8-4592-a76f-1c99f7cdf23a  e07
> UN  10.234.72.138     10.96 GB   256     42.9%
> 0d6725f0-1357-423d-85c1-153fb94257d5  e03
> UN  10.234.72.137     11.09 GB   256     45.7%
> 492190d7-3055-4167-8699-9c6560e28164  e03
> UN  10.234.72.136     11.91 GB   256     41.1%
> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466  c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host
> ID                               Rack
> UN  10.98.255.139     16.94 GB   256     43.8%
> 3523e80c-8468-4502-b334-79eabc3357f0  g10
> UN  10.98.255.138     12.62 GB   256     42.4%
> a2bcddf1-393e-453b-9d4f-9f7111c01d7f  i02
> UN  10.98.255.137     10.59 GB   256     38.4%
> f851b6ee-f1e4-431b-8beb-e7b173a77342  i02
> UN  10.98.255.136     11.89 GB   256     42.9%
> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e  a09
> UN  10.98.255.135     10.29 GB   256     40.4%
> e2d020a5-97a9-48d4-870c-d10b59858763  a09
> UN  10.98.255.134     16.19 GB   256     52.3%
> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb  h06
> UN  10.98.255.140     127.84 KB  256     39.9%
> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72  g10
>
> We can see that the new node (10.98.255.140) contains only 127,84KB.
> We saw also that there was no network traffic between the nodes.
>
> Then we added a new node in the 2nd datacenter (dc b1)
>
>
>
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host
> ID                               Rack
> UN  10.234.72.135     12.95 GB   256     42.0%
> 2fc583b2-822f-4347-9fab-5e9d10d548c9  c01
> UN  10.234.72.134     20.11 GB   256     53.1%
> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1  e01
> UN  10.234.72.140     122.25 KB  256     41.9%
> 501ea498-8fed-4cc8-a23a-c99492bc4f26  e07
> UN  10.234.72.139     20.46 GB   256     40.2%
> 95661392-ccd8-4592-a76f-1c99f7cdf23a  e07
> UN  10.234.72.138     13.21 GB   256     40.9%
> 0d6725f0-1357-423d-85c1-153fb94257d5  e03
> UN  10.234.72.137     13.34 GB   256     42.9%
> 492190d7-3055-4167-8699-9c6560e28164  e03
> UN  10.234.72.136     14.16 GB   256     39.0%
> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466  c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns (effective)  Host
> ID                               Rack
> UN  10.98.255.139     19.19 GB   256     43.8%
> 3523e80c-8468-4502-b334-79eabc3357f0  g10
> UN  10.98.255.138     14.9 GB    256     42.4%
> a2bcddf1-393e-453b-9d4f-9f7111c01d7f  i02
> UN  10.98.255.137     12.49 GB   256     38.4%
> f851b6ee-f1e4-431b-8beb-e7b173a77342  i02
> UN  10.98.255.136     14.13 GB   256     42.9%
> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e  a09
> UN  10.98.255.135     12.16 GB   256     40.4%
> e2d020a5-97a9-48d4-870c-d10b59858763  a09
> UN  10.98.255.134     18.85 GB   256     52.3%
> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb  h06
> UN  10.98.255.140     2.24 GB    256     39.9%
> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72  g10
>
>
> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
> The new node in the 1st datacenter contains now 2,24 GB because we were
> inserting data in the cluster while adding the new nodes.
>
> Then we started a repair from the new node in the 2nd datacenter :
> time nodetool repair
>
>
> We can see that the old nodes are sending data to the new node :
> root@node007:~# nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Streaming from: /10.98.255.137
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
> sections=130 progress=0/15598366 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
> sections=107 progress=0/429517 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
> sections=109 progress=0/696057 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
> sections=57 progress=0/189844 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
> sections=124 progress=56492032/4597955 - 1228%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
> sections=113 progress=0/850632 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
> sections=130 progress=0/16815423 - 0%
> Streaming from: /10.98.255.136
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
> sections=104 progress=0/3545520 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
> sections=43 progress=0/252976 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
> sections=91 progress=0/443365 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
> sections=101 progress=11337728/4188624 - 270%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
> sections=94 progress=0/758418 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
> sections=88 progress=0/396552 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
> sections=105 progress=0/12363907 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
> sections=87 progress=0/506537 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
> sections=42 progress=0/197531 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
> sections=110 progress=0/12689898 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
> sections=83 progress=0/379628 - 0%
> Streaming from: /10.98.255.134
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
> sections=108 progress=0/10715348 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
> sections=82 progress=0/8177161 - 0%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
> sections=84 progress=25613959168/8556942 - 299335%
>    hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
> sections=79 progress=0/506293 - 0%
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0            835
> Responses                       n/a         0           6765
>
> So, it looks that the repair is required if we want to add new nodes in
> our platform, but I don't understand why.
>
> Regards.
>
> Jean Armel
>
>
>
>

Mime
View raw message