cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
Date Tue, 03 Nov 2015 22:15:28 GMT


Ariel Weisberg commented on CASSANDRA-9748:

Summing up our private conversation. The use case is where someone has two interfaces they
want to use for intra-cluster traffic. One is accessible from other DCs the other is intra-DC.
Right now they can only specify one broadcast address and that has to be the one reachable
everywhere. This was fine for EC2 with it's weird private/public IP magic.

For real separate interfaces it doesn't work because you need to select between one of two
IPs. The addition of a local IP to the GossipingPropertyFile snitch means that instead of
advertising the broadcast address via gossip (as the internal interface) it can advertise
a second interface.

My question is why is this necessary at all? My ops fu is pretty weak but I thought that a
single interface is generally fine (unless there is a performance issue) because you can configure
your network to do the right thing WRT to reachability and efficiently routing traffic between
and intra DCs with a single IP.

One thing that seems suboptimal is binding to all interfaces instead of a list of interfaces.
This forces people to manually block connections on interfaces we really shouldn't be accepting
on. That said this is mechanism not policy and it enables people to do something they couldn't
before so it's an improvement and they have options outside of C*.

> Can't see other nodes when using multiple network interfaces
> ------------------------------------------------------------
>                 Key: CASSANDRA-9748
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>         Environment: Cassandra 2.0.16; multi-DC configuration
>            Reporter: Roman Bielik
>            Assignee: Paulo Motta
>              Labels: docs-impacting
>             Fix For: 2.1.x, 2.2.x, 3.0.x
>         Attachments: system_node1.log, system_node2.log
> The idea is to setup a multi-DC environment across 2 different networks based on the
following configuration recommendations:
> Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and
DC2: 10.0.2.x). The second one a "public" network where all nodes can see each other (this
one has a higher latency). 
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:*
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess). The nodetool
ring/status shows only the local node but not the other ones (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that is not the
required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp        0      0      *                   LISTEN  
> tcp        0      0     *                   LISTEN  
> tcp        0      0     *                   LISTEN  
> tcp        0      0     *                   LISTEN  
> tcp        0      0                 ESTABLISHED
> tcp        0      0                   ESTABLISHED

This message was sent by Atlassian JIRA

View raw message