cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernardino Mota <bernardino.m...@knowledgeworks.pt>
Subject Nodes fail to reconnect after several hours of network failure.
Date Thu, 21 Jan 2016 13:09:52 GMT
Using Cassandra 2.2.4 on Ubuntu.

We have a cluster with two nodes that during several hours failed to connect with each other
due to network problems. The database continued to be used in one of the nodes with writes
being stored in the Hints file as supposed.

But now that the network is OK again and each machine can communicate we see that each node
indicates the other is DOWN and does not replicates. 

When the network came up we started to see in log files "Convicting /192.168.1.102 with status
NORMAL - alive false"

It seems each node evictions each other and later failing to reconnect.

Is there some configuration that we might be missing ? Any help would be much appreciated.

 

- NODE 192.168.1.10 - "nodetool status” 

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
DN  192.168.1.102  12.02 MB   256          ?       ff906882-8224-40ac-8cdb-98f5e725814d  rack1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  192.168.1.10   41.87 MB   256          ?       51650afd-84dd-4e25-a6f0-13627858d5dc  rack1



- NODE 192.168.1.102  - “nodetool status"

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  192.168.1.102  12.4 MB    256          ?       ff906882-8224-40ac-8cdb-98f5e725814d  rack1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
DN  192.168.1.10   26.31 MB   256          ?       51650afd-84dd-4e25-a6f0-13627858d5dc  rack1



Mime
View raw message