cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Brotman" <>
Subject RE: [EXTERNAL] RE: Adding new DC?
Date Tue, 13 Mar 2018 05:24:30 GMT


Also to check:


You should use the same list of seeds, probably two in each data center if you will have five
nodes in each, in all the yaml files.  All the seeds node addresses from all the data centers
listed in each yaml file where it says “-seeds:”.  I’m not sure from your previous replies
if you’re doing that.


Let us know your results.


Kenneth Brotman


From: Kenneth Brotman [] 
Sent: Monday, March 12, 2018 7:14 PM
To: ''
Subject: RE: [EXTERNAL] RE: Adding new DC?




Sorry for asking you things you already answered.  You provided a lot of good information
and you know what you’re are doing.  It’s going to be something really simple to figure
out.  While I read through the thread more closely, I’m guessing we are right on top of
it so could I ask you:


Please read through
as it probably has the answer.  


One of things it says specifically is: 

Additional cassandra.yaml configuration for non-EC2 implementations

If multiple network interfaces are used in a non-EC2 implementation, enable thelisten_on_broadcast_address

listen_on_broadcast_address: true

In non-EC2 environments, the public address to private address routing is not automatically
enabled. Enabling listen_on_broadcast_address allows DSE to listen on both listen_address
andbroadcast_address with two network interfaces.


Please consider that specially and be sure everything else it mentions is done


You said you changed the broadcast_rpc_address in one of the instances in GCE and saw a change.
 Did you update the other nodes in GCE?  And then restarted each one (in a rolling manner)?


Did you restart each node in each datacenter starting with the seed nodes since you last updated
a yaml file?


Could the client in your application be causing the problem?  


Kenneth Brotman



From: Kunal Gangakhedkar [] 
Sent: Monday, March 12, 2018 4:43 PM
Cc: Nikhil Soman
Subject: Re: [EXTERNAL] RE: Adding new DC?


Yes, that's correct. The customer wants us to migrate the cassandra setup in their AWS account.





On 13 March 2018 at 04:56, Kenneth Brotman <> wrote:

I didn’t understand something.  Are you saying you are using one data center on Google and
one on Amazon?


Kenneth Brotman


From: Kunal Gangakhedkar [] 
Sent: Monday, March 12, 2018 4:24 PM
Cc: Nikhil Soman
Subject: Re: [EXTERNAL] RE: Adding new DC?



On 13 March 2018 at 03:28, Kenneth Brotman <> wrote:

You can’t migrate and upgrade at the same time perhaps but you could do one and then the
other so as to end up on new version.  I’m guessing it’s an error in the yaml file or
a port not open.  Is there any good reason for a production cluster to still be on version


I'm not trying to migrate AND upgrade at the same time. However, the apt repo shows only 2.120
as the available version.

This is the output from the new node in AWS


ubuntu@ip-10-0-43-213:~$ apt-cache policy cassandra 
 Installed: 2.1.20 
 Candidate: 2.1.20 
 Version table: 
*** 2.1.20 500 
       500 21x/main amd64 Packages 
       100 /var/lib/dpkg/status

Regarding open ports, I can cqlsh into the GCE node(s) from the AWS node into GCE nodes.

As I mentioned earlier, I've opened the ports 9042, 7000, 7001 in GCE firewall for the public
IP of the AWS instance.


I mentioned earlier - there are some differences in the column types - for example, date (>=
2.2) vs. timestamp (2.1.x)

The application has not been updated yet.

Hence sticking to 2.1.x for now.


And, so far, 2.1.x has been serving the purpose.




Kenneth Brotman


From: Durity, Sean R [] 
Sent: Monday, March 12, 2018 11:36 AM
Subject: RE: [EXTERNAL] RE: Adding new DC?


You cannot migrate and upgrade at the same time across major versions. Streaming is (usually)
not compatible between versions.


As to the migration question, I would expect that you may need to put the external-facing
ip addresses in several places in the cassandra.yaml file. And, yes, it would require a restart.
Why is a non-restart more desirable? Most Cassandra changes require a restart, but you can
do a rolling restart and not impact your application. This is fairly normal admin work and
can/should be automated.


How large is the cluster to migrate (# of nodes and size of data). The preferred method might
depend on how much data needs to move. Is any application outage acceptable?


Sean Durity

lord of the (C*) rings (Staff Systems Engineer – Cassandra)

From: Kunal Gangakhedkar [] 
Sent: Sunday, March 11, 2018 10:20 PM
Subject: [EXTERNAL] RE: Adding new DC?


Hi Kenneth,


Replies inline below.


On 12-Mar-2018 3:40 AM, "Kenneth Brotman" <> wrote:

Hi Kunal,


That version of Cassandra is too far before me so I’ll let others answer.  I was wonder
why you wouldn’t want to end up on 3.0x if you’re going through all the trouble of migrating



Application side constraints - some data types are different between 2.1.x and 3.x (for example,
date vs. timestamp).


Besides, this is production setup - so, cannot take risk

Are both data centers in the same region on AWS?  Can you provide yaml file for us to see?



No, they are in different regions - GCE setup is in us-east while AWS setup is in Asia-south




Kenneth Brotman


From: Kunal Gangakhedkar [] 
Sent: Sunday, March 11, 2018 2:32 PM
Subject: Adding new DC?


Hi all,


We currently have a cluster in GCE for one of the customers.

They want it to be migrated to AWS.


I have setup one node in AWS to join into the cluster by following:


Will add more nodes once the first one joins successfully.


The node in AWS has an elastic IP - which is white-listed for ports 7000-7001, 7199, 9042
in GCE firewall


The snitch is set to GossipingPropertyFileSnitch. The GCE setup has dc=DC1, rack=RAC1 while
on AWS, I changed the DC to dc=DC2.


When I start cassandra service on the AWS instance, I see the version handshake msgs in the
logs trying to connect to the public IPs of the GCE nodes: - Handshaking version with /xx.xxxx.xx

However, nodetool status output on both sides don't show the other side at all. That is, the
GCE setup doesn't show the new DC (dc=DC2) and the AWS setup doesn't show old DC (dc=DC1).


In cassandra.yaml file, I'm only using listen_interface and rpc_interface settings - no explicit
IP addresses used - so, ends up using the internal private IP ranges.


Do I need to explicitly add the broadcast_address? for both side?

Would that require restarting of cassandra service on GCE side? Or is it possible to change
that setting on-the-fly without a restart?


I would prefer a non-restart option.


PS: The cassandra version running in GCE is 2.1.18 while the new node setup in AWS is running
2.1.20 - just in case if that's relevant







The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.



View raw message