incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Armel Luce <jaluc...@gmail.com>
Subject Re: Adding new nodes in a cluster with virtual nodes
Date Thu, 28 Feb 2013 08:35:42 GMT
Hi Aaron,

I tried again to add a node in the cluster. This time, I added the new node
in the seeds list after the bootstrap (the first time, I added the new node
in the seeds list before the bootstrap).

And it works !!!


Thanks Aaron.

Regards.

Jean Armel.

2013/2/22 Jean-Armel Luce <jaluce06@gmail.com>

> Thanks Aaron.
>
> I shall investigate more next week about this.
>
> regards.
>
> Jean Armel
>
>
> 2013/2/22 aaron morton <aaron@thelastpickle.com>
>
>> So, it looks that the repair is required if we want to add new nodes in
>> our platform, but I don't understand why.
>>
>> Bootstrapping should take care of it. But new seed nodes do not
>> bootstrap. Check the logs on the nodes you added to see what messages have
>> "bootstrap" in them.
>>
>> Anytime you are worried about things like this throw in a nodetool
>> repair. If you are using QUOURM for read and writes you will still be
>> getting consistent data, so long as you have only added one node. Or one
>> node every RF'th nodes.
>>
>> Cheers
>>
>>
>>    -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 22/02/2013, at 9:55 PM, Jean-Armel Luce <jaluce06@gmail.com> wrote:
>>
>> Hi Aaron,
>>
>> Thanks for your answer.
>>
>>
>> I apologize, I did a mistake in my 1st mail. The cluster was only 12
>> nodes instead of 16 (it is a test cluster).
>> There are 2 datacenters b1 and s1.
>>
>> Here is the result of nodetool status after adding a new node in the 1st
>> datacenter (dc s1):
>> root@node007:~# nodetool status
>> Datacenter: b1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns (effective)  Host
>> ID                               Rack
>> UN  10.234.72.135     10.71 GB   256     44.6%
>> 2fc583b2-822f-4347-9fab-5e9d10d548c9  c01
>> UN  10.234.72.134     16.74 GB   256     63.7%
>> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1  e01
>> UN  10.234.72.139     17.09 GB   256     62.0%
>> 95661392-ccd8-4592-a76f-1c99f7cdf23a  e07
>> UN  10.234.72.138     10.96 GB   256     42.9%
>> 0d6725f0-1357-423d-85c1-153fb94257d5  e03
>> UN  10.234.72.137     11.09 GB   256     45.7%
>> 492190d7-3055-4167-8699-9c6560e28164  e03
>> UN  10.234.72.136     11.91 GB   256     41.1%
>> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466  c01
>> Datacenter: s1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns (effective)  Host
>> ID                               Rack
>> UN  10.98.255.139     16.94 GB   256     43.8%
>> 3523e80c-8468-4502-b334-79eabc3357f0  g10
>> UN  10.98.255.138     12.62 GB   256     42.4%
>> a2bcddf1-393e-453b-9d4f-9f7111c01d7f  i02
>> UN  10.98.255.137     10.59 GB   256     38.4%
>> f851b6ee-f1e4-431b-8beb-e7b173a77342  i02
>> UN  10.98.255.136     11.89 GB   256     42.9%
>> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e  a09
>> UN  10.98.255.135     10.29 GB   256     40.4%
>> e2d020a5-97a9-48d4-870c-d10b59858763  a09
>> UN  10.98.255.134     16.19 GB   256     52.3%
>> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb  h06
>> UN  10.98.255.140     127.84 KB  256     39.9%
>> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72  g10
>>
>> We can see that the new node (10.98.255.140) contains only 127,84KB.
>> We saw also that there was no network traffic between the nodes.
>>
>> Then we added a new node in the 2nd datacenter (dc b1)
>>
>>
>>
>> root@node007:~# nodetool status
>> Datacenter: b1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns (effective)  Host
>> ID                               Rack
>> UN  10.234.72.135     12.95 GB   256     42.0%
>> 2fc583b2-822f-4347-9fab-5e9d10d548c9  c01
>> UN  10.234.72.134     20.11 GB   256     53.1%
>> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1  e01
>> UN  10.234.72.140     122.25 KB  256     41.9%
>> 501ea498-8fed-4cc8-a23a-c99492bc4f26  e07
>> UN  10.234.72.139     20.46 GB   256     40.2%
>> 95661392-ccd8-4592-a76f-1c99f7cdf23a  e07
>> UN  10.234.72.138     13.21 GB   256     40.9%
>> 0d6725f0-1357-423d-85c1-153fb94257d5  e03
>> UN  10.234.72.137     13.34 GB   256     42.9%
>> 492190d7-3055-4167-8699-9c6560e28164  e03
>> UN  10.234.72.136     14.16 GB   256     39.0%
>> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466  c01
>> Datacenter: s1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns (effective)  Host
>> ID                               Rack
>> UN  10.98.255.139     19.19 GB   256     43.8%
>> 3523e80c-8468-4502-b334-79eabc3357f0  g10
>> UN  10.98.255.138     14.9 GB    256     42.4%
>> a2bcddf1-393e-453b-9d4f-9f7111c01d7f  i02
>> UN  10.98.255.137     12.49 GB   256     38.4%
>> f851b6ee-f1e4-431b-8beb-e7b173a77342  i02
>> UN  10.98.255.136     14.13 GB   256     42.9%
>> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e  a09
>> UN  10.98.255.135     12.16 GB   256     40.4%
>> e2d020a5-97a9-48d4-870c-d10b59858763  a09
>> UN  10.98.255.134     18.85 GB   256     52.3%
>> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb  h06
>> UN  10.98.255.140     2.24 GB    256     39.9%
>> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72  g10
>>
>>
>> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
>> The new node in the 1st datacenter contains now 2,24 GB because we were
>> inserting data in the cluster while adding the new nodes.
>>
>> Then we started a repair from the new node in the 2nd datacenter :
>> time nodetool repair
>>
>>
>> We can see that the old nodes are sending data to the new node :
>> root@node007:~# nodetool netstats
>> Mode: NORMAL
>> Not sending any streams.
>> Streaming from: /10.98.255.137
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
>> sections=130 progress=0/15598366 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
>> sections=107 progress=0/429517 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
>> sections=109 progress=0/696057 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
>> sections=57 progress=0/189844 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
>> sections=124 progress=56492032/4597955 - 1228%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
>> sections=113 progress=0/850632 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
>> sections=130 progress=0/16815423 - 0%
>> Streaming from: /10.98.255.136
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
>> sections=104 progress=0/3545520 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
>> sections=43 progress=0/252976 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
>> sections=91 progress=0/443365 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
>> sections=101 progress=11337728/4188624 - 270%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
>> sections=94 progress=0/758418 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
>> sections=88 progress=0/396552 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
>> sections=105 progress=0/12363907 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
>> sections=87 progress=0/506537 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
>> sections=42 progress=0/197531 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
>> sections=110 progress=0/12689898 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
>> sections=83 progress=0/379628 - 0%
>> Streaming from: /10.98.255.134
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
>> sections=108 progress=0/10715348 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
>> sections=82 progress=0/8177161 - 0%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
>> sections=84 progress=25613959168/8556942 - 299335%
>>    hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
>> sections=79 progress=0/506293 - 0%
>> Pool Name                    Active   Pending      Completed
>> Commands                        n/a         0            835
>> Responses                       n/a         0           6765
>>
>> So, it looks that the repair is required if we want to add new nodes in
>> our platform, but I don't understand why.
>>
>> Regards.
>>
>> Jean Armel
>>
>>
>>
>>
>

Mime
View raw message