incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Wintle <timwin...@gmail.com>
Subject Re: Populating seeds dynamically
Date Fri, 07 Jun 2013 00:07:44 GMT
On Mon, 2013-06-03 at 17:20 -0700, Aiman Parvaiz wrote:
> @Faraaz check out the comment by Aaron morton here : http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Seed-Nodes-td6077958.html
> Having same nodes is a good idea but it is not necessary.
> > In your case, sure the nodes will be in the cluster for 10
> > minutes but what about sporadic failures that cause them to leave the ring and
> > then re-enter it? At that point, you might reach the network fragmentation
> > issue.
> 
> I am not sure I understand this completely, if a node leaves the ring and re enters it
would use its seed nodes to know about the ring and these nodes would be the ones which are
part of ring so I don't see any information lag happening here.

As I understand the risk:

If you start with four nodes in the ring:

10.1.1.1
10.1.1.2
10.2.1.1
10.2.1.2

.. Then the connection between 10.1.0.0/16 and 10.2.0.0/16 is lost
temporarily through some routing issue / firewall config etc.

You run puppet to re-deploy on 10.1.1.1 and 10.1.1.2 . When they
discover nodes on the current ring, they only find the 10.1.0.0 nodes.

You do the same on 10.2.1.1 and 10.2.1.2, and they only find the nodes
on that segment of the ring (I'm not used to puppet - if the config
generation is run once on a single machine then you could still end up
with that by chance assuming that machine can still reach both parts of
the cluster)

At this point your networking team spot the routing issue and fix it -
but it's too late.

So you now have two disjoint rings running, which don't have any of the
other subring nodes in their seed nodes respectively.

.. all that is fairly unlikely to happen, but it's possible - and the
more switches and machines that are involved the more likely it becomes
that it will happen.

I'm not sure if that's the only risk - but if it is then IMHO it's a
question of if that risk is acceptable, and if so then adding monitoring
to continually check for that situation.

Tim


Mime
View raw message