cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hannu Kröger <hkro...@gmail.com>
Subject Re: Node replacement strategy with AWS EBS
Date Wed, 14 Jun 2017 18:55:19 GMT
Hi,

So, if it works, great.

auto_bootstrap false is not needed when you have system keyspace as also
mentioned in the article. Now you are likely to have different tokens then
the previous node (unless those were manually configured to match the old
node) and repair and cleanup are needed to get that node to “right” state.
But if the tokens were configured ok then repair & cleanup are not needed.

Cheers,
Hannu

On 14 June 2017 at 13:37:29, Rutvij Bhatt (rutvij@sense.com) wrote:

Thanks again for your help! To summarize for anyone who stumbles onto this
in the future, this article covers the procedure well:
https://www.eventbrite.com/engineering/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/

It is more or less what Hannu suggested.

I carried out the following steps:
1. safely stop the cassandra instance (nodetool drain + service cassandra
stop)
2. Shut down the ec2 instance.
3. detach the storage volume from old instance.
4. attach to new instance.
5. point cassandra configuration on new instance to this drive and set
auto_bootstrap: false
6. start cassandra on new instance. Once it has established connection with
peers, you will notice that it takes over the token ranges on its own.
Doing a select on the system.peers table will show that the old node is
gone.
7. Run nodetool repair if need be.

On Tue, Jun 13, 2017 at 1:01 PM Rutvij Bhatt <rutvij@sense.com> wrote:

> Nevermind, I misunderstood the first link. In this case, the replacement
> would just be leaving the listen_address as is (to
> InetAddress.getLocalHost()) and just start the new instance up as you
> pointed out in your original answer Hannu.
>
> Thanks.
>
> On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt <rutvij@sense.com> wrote:
>
>> Hannu/Nitan,
>>
>> Thanks for your help so far! From what you said in your first response, I
>> can get away with just attaching the EBS volume to Cassandra and starting
>> it with the old node's private IP as my listen_address because it will take
>> over the token assignment from the old node using the data files? With
>> regards to "Cassandra automatically realizes that have just effectively
>> changed IP address.", it says in the first link to change this manually to
>> the desired address - does this not apply in my case if I'm replacing the
>> old node?
>>
>> As for the plan I outlined earlier, is this more for DR scenarios where I
>> have lost a node due to hardware failure and I need to recover the data in
>> a safe manner by requesting a stream from the other replicas?  Am I
>> understanding this right?
>>
>>
>> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger <hkroger@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> So the local information about tokens is stored in the system keyspace.
>>> Also the host id and all that.
>>>
>>> Also documented here:
>>>
>>> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>>>
>>> If for any reason that causes issues, you can also check this:
>>> https://issues.apache.org/jira/browse/CASSANDRA-8382
>>>
>>> If you copy all cassandra data, you are on the safe side. Good point in
>>> the links is that if you have IP addresses in topolgy or other files, then
>>> update those as well.
>>>
>>> Hannu
>>>
>>> On 13 June 2017 at 11:53:13, Nitan Kainth (nitan@bamlabs.com) wrote:
>>>
>>> Hannu,
>>>
>>> "Cassandra automatically realizes that have just effectively changed IP
>>> address” —> are you sure C* will take care of IP change as is? How will
it
>>> know which token range to be assigned to this new IP address?
>>>
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger <hkroger@gmail.com> wrote:
>>>
>>> Cassandra automatically realizes that have just effectively changed IP
>>> address
>>>
>>>
>>>

Mime
View raw message