cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Girish Kamarthi <girish.kamar...@stellapps.com>
Subject Cassandra 3 node cluster with intermittent network issues on one node
Date Mon, 03 Oct 2016 06:38:53 GMT
Hi All,

I want to test out a scenario where there is intermittent network issues on
one of the node.

I've got Cassandra 3.7 cluster of 3 nodes with the keyspace replication
factor of 3.

All the 3 nodes(node A, node B, node C) are started and are in sync. When
one of the cassandra node went down (node A), I restarted cassandra, the
node A gets in sync with the other nodes B & C.

Now my question is when one of the node has issues like intermittent
network issues (cassandra is still up and running). Say node A is having
network issues, the nodetool status on the other 2 nodes b & C shows that
the node A is down.

*Debug.log of Node B & C:*

DEBUG [GossipTasks:1] 2016-10-03 11:46:18,922 Gossiper.java:337 -
Convicting /10.1.1.4 with status NORMAL - alive false

When the network is back on the node A the nodetool status shows that the
other nodes are down.

*Debug.log of Node A:*

DEBUG [GossipTasks:1] 2016-10-03 11:47:23,613 Gossiper.java:337 -
Convicting /10.1.1.5 with status NORMAL - alive false

DEBUG [GossipTasks:1] 2016-10-03 11:47:23,614 Gossiper.java:337 -
Convicting /10.1.1.6 with status NORMAL - alive false


Below are the configuration changes I made in the cassandra.yaml files.

Node 01

cluster_name: 'Test Cluster'

num_tokens: 256

seed_provider: - class_name:
org.apache.cassandra.locator.SimpleSeedProvider

parameters: - seeds: "10.1.1.4,10.1.1.5,10.1.1.6"

listen_address: 10.1.1.4

broadcast_address: 10.1.1.4

rpc_address: 0.0.0.0

broadcast_rpc_address: 10.1.1.4


Node02

cluster_name: 'Test Cluster'

num_tokens: 256

seed_provider: - class_name:
org.apache.cassandra.locator.SimpleSeedProvider

parameters: - seeds: "10.1.1.4,10.1.1.5,10.1.1.6"

listen_address: 10.1.1.5

broadcast_address: 10.1.1.5

rpc_address: 0.0.0.0

broadcast_rpc_address: 10.1.1.5


Node03

cluster_name: 'Test Cluster'

num_tokens: 256

seed_provider: - class_name:
org.apache.cassandra.locator.SimpleSeedProvider

parameters: - seeds: "10.1.1.4,10.1.1.5,10.1.1.6"

listen_address: 10.1.1.6

broadcast_address: 10.1.1.6

rpc_address: 0.0.0.0

broadcast_rpc_address: 10.1.1.6


Nodetool status on node A when the network is up shows that the other nodes
are down (DN).

Nodetool status on the other nodes B & C shows that the node 1 is down (DN)

How does the handshaking works in this scenario?

Why the node A is not in sync with the other nodes when the network is up?

Please give me some inputs on resolving this issue.

Thanks & Regards,
Girish Kumar Kamarthi
+91-9986427891

Mime
View raw message