Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of chris.burroughs@gmail.com
 designates 209.85.213.180 as permitted sender)
Message-ID: <51BF5C79.7020905@gmail.com>
Date: Mon, 17 Jun 2013 14:59:05 -0400
From: Chris Burroughs <chris.burroughs@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130510 Thunderbird/17.0.6
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: multi-dc clusters with 'local' ips and no vpn
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Cassandra makes the totally reasonable assumption that the entire
cluster is in one routable address space.  We unfortunately had a
situation where:
 * nodes can talk to each other in the same dc on an internal address,
but not talk to each other over their external 1:1 NAT address.
 * nodes can talk to nodes in the other dc over the external address,
but there is no usable shared internal address space they can talk over

In case anyone else finds themselves in the same situation we have what
we think is a working solution in pre-production.  CASSANDRA-5630
handles the "reconnect trick" to prefer the local ip when in the same
DC.  And some iptables rules allow the local nodes to do the initial
gossiping with each other before that switch.

for each node in same dc:
    'iptables -t nat -A OUTPUT -j DNAT -p tcp --dst %s --dport 7000 -o
eth0  --to-destination %s' % (ext_ip, local_ip)