Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1831C1867C for ; Wed, 12 Aug 2015 03:45:11 +0000 (UTC) Received: (qmail 50712 invoked by uid 500); 12 Aug 2015 03:45:07 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 50671 invoked by uid 500); 12 Aug 2015 03:45:07 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50661 invoked by uid 99); 12 Aug 2015 03:45:07 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2015 03:45:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 3AC50C0DF9 for ; Wed, 12 Aug 2015 03:45:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id B1jq_J8zFF_A for ; Wed, 12 Aug 2015 03:44:54 +0000 (UTC) Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id BE92B42B7E for ; Wed, 12 Aug 2015 03:44:53 +0000 (UTC) Received: by iodt126 with SMTP id t126so8085189iod.2 for ; Tue, 11 Aug 2015 20:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=VtpO8DQ1DunLlSTwy0vd2OYWFjnE+rXdGxxpx7NTh+o=; b=CNzizZr8bjtBsB2Hh3RZd+ZS1X2Q/+uu2+EYnML/MtjYVEO2NKxolY7Wj5ukzxK/MH dFJHQ04QAXPpCJ+DdNEPkig6GWo75bxuZ6BbXP//H31CK3kW/2otv2xvDEjz3DzYIXNN aIfgo2h65WC1mtiVO67aShdmIhjnLmgOxfr2y4lCfR5MjvSKz8mbhSsW/ugB6qc9m7Ov /s4Rtm0QRyGzsvy+KB9dv9ztxUqEZxmF+MKAjZD83UB1gf+uYB5PHod6tO+jKflmueBf U6x/tdHutXGqYmaHblZwuLqGaPbIqm7VN/d4/OTjwCNWPoJoKvvCP4rOEp9tBP9jbBHq AjmQ== MIME-Version: 1.0 X-Received: by 10.107.163.11 with SMTP id m11mr30856367ioe.31.1439351093352; Tue, 11 Aug 2015 20:44:53 -0700 (PDT) Received: by 10.79.15.199 with HTTP; Tue, 11 Aug 2015 20:44:53 -0700 (PDT) In-Reply-To: References: Date: Tue, 11 Aug 2015 23:44:53 -0400 Message-ID: Subject: Re: Change from single region EC2 to multi-region From: Asher Newcomer To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11402ad8a06213051d150a74 --001a11402ad8a06213051d150a74 Content-Type: text/plain; charset=UTF-8 Thank you all for the help and ideas. In the end, this was a configuration issue in AWS, and not an issue with Cassandra. Regards On Tue, Aug 11, 2015 at 7:26 PM, Bryan Cheng wrote: > broadcast_address to public ip should be the correct configuration. > Assuming your firewall rules are all kosher, you may need to clear gossip > state? > http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_gossip_purge.html > > > ---------- Forwarded message ---------- > From: Asher Newcomer > Date: Tue, Aug 11, 2015 at 11:51 AM > Subject: Change from single region EC2 to multi-region > To: user@cassandra.apache.org > > > X-post w/ SO: link > > > I have (had) a working 4 node Cassandra cluster setup in an EC2 VPC. Setup > was as follows: > > 172.18.100.110 - seed - DC1 / RAC1 > > 172.18.100.111 - DC1 / RAC1 > > 172.18.100.112 - seed - DC1 / RAC2 > > 172.18.100.113 - DC1 / RAC2 > > All of the above nodes are in East-1D, and I have configured it using the > GossipingPropertyFileSnitch (I would rather not use the EC2 specific > snitches). > > listen_address & broadcast_address were both set to the node's private IP. > > I then wanted to expand the cluster into a new region (us-west). Because > cross-region private IP communication is not supported in EC2, I attempted > to change the settings to have the nodes communicate through their public > IPs. > > listen_address remained set to private IP > broadcast_address was changed to the public IP > seeds_list IPs were changed to the appropriate public IPs > > I restarted the nodes one by one expecting them to simply 'work', but now > they only see themselves and not the other nodes. > > nodetool status consistently returns: > > Datacenter: DC1 > =============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID Rack > DN 172.18.100.112 ? 256 ? 968aaa8a-32b7-4493-9747-3df1c3784164 r1 > DN 172.18.100.113 ? 256 ? 8e03643c-9db8-4906-aabc-0a8f4f5c087d r1 > UN [public IP of local node] 75.91 GB 256 ? > 6fdcc85d-6c78-46f2-b41f-abfe1c86ac69 RAC1 > DN 172.18.100.110 ? 256 ? fb7b78a8-d1cc-46fe-ab18-f0d3075cb426 r1 > > On each individual node, the other nodes seem 'stuck' using the private IP > addresses. > > *How do I force the nodes to look for each other at their public > addresses?* > > I have fully opened the EC2 security group/firewall as a test to rule out > any problems there - and it hasn't helped. > > Any ideas most appreciated. > > --001a11402ad8a06213051d150a74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you all for the help and ideas.

I= n the end, this was a configuration issue in AWS, and not an issue with Cas= sandra.

= Regards

On T= ue, Aug 11, 2015 at 7:26 PM, Bryan Cheng <bryan@blockcypher.com>= ; wrote:
bro= adcast_address to public ip should be the correct configuration. Assuming y= our firewall rules are all kosher, you may need to clear gossip state?=C2= =A0http://docs.datastax.com/en/cass= andra/2.0/cassandra/operations/ops_gossip_purge.html


---------- Forwarded message --= --------
From: Asher Newcomer <asher= new@gmail.com>
Date: Tue, Aug 11, 2015 at 11:51 AM
Subj= ect: Change from single region EC2 to multi-region
To: user@cassandra.apache.org=


X-post w/ SO:=C2=A0link

I ha= ve (had) a working 4 node Cassandra cluster setup in an EC2 VPC. Setup was = as follows:

172.18.100.110 - seed - DC1 / RAC1

172.18.100.111 - DC1 / RAC1

172.18.100.112 - seed - = DC1 / RAC2

172.18.100.113 - DC1 / RAC2

Al= l of the above nodes are in East-1D, and I have configured it using the Gos= sipingPropertyFileSnitch (I would rather not use the EC2 specific snitches)= .

listen_address & broadcast_address were both set to = the node's private IP.

I then wanted to expand the clu= ster into a new region (us-west). Because cross-region private IP communica= tion is not supported in EC2, I attempted to change the settings to have th= e nodes communicate through their public IPs.

listen_addre= ss remained set to private IP
broadcast_address was changed to the publi= c IP
seeds_list IPs were changed to the appropriate public IPs

I restarted the nodes one by one expecting them to simply '= work', but now they only see themselves and not the other nodes.

nodetool status consistently returns:

Datacente= r: DC1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down=
|/ State=3DNormal/Leaving/Joining/Moving
-- Address Load Tokens Owns= Host ID Rack
DN 172.18.100.112 ? 256 ? 968aaa8a-32b7-4493-9747-3df1c378= 4164 r1
DN 172.18.100.113 ? 256 ? 8e03643c-9db8-4906-aabc-0a8f4f5c087d r= 1
UN [public IP of local node] 75.91 GB 256 ? 6fdcc85d-6c78-46f2-b41f-ab= fe1c86ac69 RAC1
DN 172.18.100.110 ? 256 ? fb7b78a8-d1cc-46fe-ab18-f0d307= 5cb426 r1

On each individual node, the other nodes see= m 'stuck' using the private IP addresses.=C2=A0

How do I force the nodes to look for each other at their public addresses?=

I have fully opened the EC2 security group/firewa= ll as a test to rule out any problems there - and it hasn't helped.

=

Any ideas most appreciated.



--001a11402ad8a06213051d150a74--