Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1CAE718C1E for ; Tue, 11 Aug 2015 23:27:23 +0000 (UTC) Received: (qmail 43116 invoked by uid 500); 11 Aug 2015 23:27:20 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 43073 invoked by uid 500); 11 Aug 2015 23:27:20 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 43063 invoked by uid 99); 11 Aug 2015 23:27:20 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2015 23:27:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id EFF8818193D for ; Tue, 11 Aug 2015 23:27:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id gPhufdETu-na for ; Tue, 11 Aug 2015 23:27:06 +0000 (UTC) Received: from mail-ob0-f180.google.com (mail-ob0-f180.google.com [209.85.214.180]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 9135C20B86 for ; Tue, 11 Aug 2015 23:27:05 +0000 (UTC) Received: by obbfr1 with SMTP id fr1so586889obb.1 for ; Tue, 11 Aug 2015 16:26:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=KuZPKJJvX+cLItl1dW1Dlmot7zWZjtlgRyDS/ShdX1Q=; b=SggLMynN+AJjw6WZ4Uju1kUR84ha1+9DbuBIHLEQFGEAKqg4w1aJHd3I7AljVQe+Ux KFMtCf4l/uWgyqWOiTmToiLt4S5gXCfN6AiCyhlJmCL0RkiZ2LycX/fEte7dJXHEOGI2 MIkA4dx10bxJHOjuDXjOr9i+8a14lmJxZ1uNH4H9dlLimQnDEMQ9Llb5aT7I6ute2Hc+ h/QJouq2kxpMuDhz3E5lqFNddIAQbBQK0m4bigFUf+NyWmNQppQSxjhcQE9L2jhI3rph LM9mUdXlr0CXs4S2+BKuvOkhs2VwhRbDzsc7dAxwxwN6TG4qw2c06/35ZNFGdi4xvvgm 3VvQ== X-Gm-Message-State: ALoCoQkJYCe78zimTu5kbTNSV7StWCLQM94TZaQaxbXKoktXPwM5LcAKRPQviIwGo2aZLi2xE0je MIME-Version: 1.0 X-Received: by 10.60.78.230 with SMTP id e6mr28693147oex.24.1439335617824; Tue, 11 Aug 2015 16:26:57 -0700 (PDT) Received: by 10.182.140.51 with HTTP; Tue, 11 Aug 2015 16:26:57 -0700 (PDT) In-Reply-To: References: Date: Tue, 11 Aug 2015 16:26:57 -0700 Message-ID: Subject: Re: Change from single region EC2 to multi-region From: Bryan Cheng To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0111b2a236c540051d117071 --089e0111b2a236c540051d117071 Content-Type: text/plain; charset=UTF-8 broadcast_address to public ip should be the correct configuration. Assuming your firewall rules are all kosher, you may need to clear gossip state? http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_gossip_purge.html ---------- Forwarded message ---------- From: Asher Newcomer Date: Tue, Aug 11, 2015 at 11:51 AM Subject: Change from single region EC2 to multi-region To: user@cassandra.apache.org X-post w/ SO: link I have (had) a working 4 node Cassandra cluster setup in an EC2 VPC. Setup was as follows: 172.18.100.110 - seed - DC1 / RAC1 172.18.100.111 - DC1 / RAC1 172.18.100.112 - seed - DC1 / RAC2 172.18.100.113 - DC1 / RAC2 All of the above nodes are in East-1D, and I have configured it using the GossipingPropertyFileSnitch (I would rather not use the EC2 specific snitches). listen_address & broadcast_address were both set to the node's private IP. I then wanted to expand the cluster into a new region (us-west). Because cross-region private IP communication is not supported in EC2, I attempted to change the settings to have the nodes communicate through their public IPs. listen_address remained set to private IP broadcast_address was changed to the public IP seeds_list IPs were changed to the appropriate public IPs I restarted the nodes one by one expecting them to simply 'work', but now they only see themselves and not the other nodes. nodetool status consistently returns: Datacenter: DC1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack DN 172.18.100.112 ? 256 ? 968aaa8a-32b7-4493-9747-3df1c3784164 r1 DN 172.18.100.113 ? 256 ? 8e03643c-9db8-4906-aabc-0a8f4f5c087d r1 UN [public IP of local node] 75.91 GB 256 ? 6fdcc85d-6c78-46f2-b41f-abfe1c86ac69 RAC1 DN 172.18.100.110 ? 256 ? fb7b78a8-d1cc-46fe-ab18-f0d3075cb426 r1 On each individual node, the other nodes seem 'stuck' using the private IP addresses. *How do I force the nodes to look for each other at their public addresses?* I have fully opened the EC2 security group/firewall as a test to rule out any problems there - and it hasn't helped. Any ideas most appreciated. --089e0111b2a236c540051d117071 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
broadcast_address to public ip should be the correct = configuration. Assuming your firewall rules are all kosher, you may need to= clear gossip state?=C2=A0http://docs.datastax.com/en= /cassandra/2.0/cassandra/operations/ops_gossip_purge.html

<= div class=3D"gmail_quote">---------- Forwarded message ----------
From: = Asher Newcomer <ashernew@gmail.com>
Date= : Tue, Aug 11, 2015 at 11:51 AM
Subject: Change from single region EC2 t= o multi-region
To: user@cas= sandra.apache.org


X-post w/ SO:=C2=A0link

I have (had) a working 4 node Cassandra cluster setup in a= n EC2 VPC. Setup was as follows:

172.18.100.110 - seed - D= C1 / RAC1

172.18.100.111 - DC1 / RAC1

172= .18.100.112 - seed - DC1 / RAC2

172.18.100.113 - DC1 / RAC= 2

All of the above nodes are in East-1D, and I have config= ured it using the GossipingPropertyFileSnitch (I would rather not use the E= C2 specific snitches).

listen_address & broadcast_addr= ess were both set to the node's private IP.

I then wan= ted to expand the cluster into a new region (us-west). Because cross-region= private IP communication is not supported in EC2, I attempted to change th= e settings to have the nodes communicate through their public IPs.

listen_address remained set to private IP
broadcast_address was= changed to the public IP
seeds_list IPs were changed to the appropriate= public IPs

I restarted the nodes one by one expecting= them to simply 'work', but now they only see themselves and not th= e other nodes.

nodetool status consistently returns:

Datacenter: DC1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
-- A= ddress Load Tokens Owns Host ID Rack
DN 172.18.100.112 ? 256 ? 968aaa8a-= 32b7-4493-9747-3df1c3784164 r1
DN 172.18.100.113 ? 256 ? 8e03643c-9db8-4= 906-aabc-0a8f4f5c087d r1
UN [public IP of local node] 75.91 GB 256 ? 6fd= cc85d-6c78-46f2-b41f-abfe1c86ac69 RAC1
DN 172.18.100.110 ? 256 ? fb7b78a= 8-d1cc-46fe-ab18-f0d3075cb426 r1

On each individual no= de, the other nodes seem 'stuck' using the private IP addresses.=C2= =A0

How do I force the nodes to look for each other at = their public addresses?

I have fully opened the EC= 2 security group/firewall as a test to rule out any problems there - and it= hasn't helped.

Any ideas most appreciated.

<= /div>

--089e0111b2a236c540051d117071--