cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa"<jji...@apache.org>
Subject Re: Disconnecting two data centers
Date Thu, 09 Mar 2017 04:07:24 GMT


On 2017-03-08 07:57 (-0800), Chuck Reynolds <creynolds@ancestry.com> wrote: 
> I was hoping I could do the following
> 
> ·         Change seeds

Definitely. 

> 
> ·         Change the topology back to simply
> 

Not necessary, can just remove the "other" datacenter from the replication strategy.

> ·         Stop nodes in datacenter 2
> 
> ·         Remove nodes in datacenter 2
> 

>From where? system.peers? 

> ·         Restart nodes in datacenter 2

Then dc2 connects back to dc1 and you end up messed up again. BOTH SIDES are going to try
to reconnect, and it's going to be ugly.

If you used an internode authenticator, you could make that lock out the other cluster.
If you used a property file snitch (NOT GPFS, plain old PFS), you could remove the other datacenter
from each topology file.
You can use a firewall.

You'll defintiely need to change seeds, and probably need to stop most of the nodes all at
once to enable this.

> 
> Somehow Cassandra holds on to the information about who was in the cluster.
> 

system.peers keeps a list of all members in the cluster. System is local strategy (not replicated),
so each node has its own copy, so IF you try hacking at it, you MUST hack at it on all nodes
(basically at the same time, because they'll repopulate via gossip).

> What if I also changed the cluster name in the Cassandra.yaml before restarting?
> 

Changing the name is very difficult (decidedly nontrivial).


Given that you need to stop all the nodes to do this (downtime), I'd be pretty tempted to
tell you to nuke one of the clusters, and use sstableloader to repopulate it as a brand new
cluster after you've nuked it. That is - Don't try to split a cluster in half, create it,
then kill half, then make a new cluster, and use sstableloader to repopulate it quickly.



Mime
View raw message