cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Roth (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-12405) node health status inconsistent
Date Mon, 08 Aug 2016 07:35:20 GMT
Benjamin Roth created CASSANDRA-12405:
-----------------------------------------

             Summary: node health status inconsistent
                 Key: CASSANDRA-12405
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12405
             Project: Cassandra
          Issue Type: Bug
         Environment: Cassandra 3.9, Linux Xenial
            Reporter: Benjamin Roth


ATM we run a 4 node cluster with cassandra 3.9
Due to another issue (hanging repairs) I am forced to restart nodes from time to time. Before
I restart a node, all nodes are listed as UP from any other node.
When I restart one node in the cluster, the health statuses of other nodes are affected as
well.

After having restarted node "cas", the "nodetool status" output on all nodes looks like this
during startup phase of cas1: https://gist.github.com/brstgt/9be77470814d2fd160617a1c06579804

After cas1 is up again, I restart cas2. During the startup phase of cas2 the status looks
like this:
https://gist.github.com/brstgt/d27ef540b2389b3a7d2d015ab83af547

The nodetool output goes along with log messages like this:
2016-08-08T07:30:06+00:00 cas1 [GossipTasks: 1] org.apache.cassandra.gms.Gossiper Convicting
/10.23.71.3 with status NORMAL - alive false
2016-08-08T07:30:06+00:00 cas1 [GossipTasks: 1] org.apache.cassandra.gms.Gossiper Convicting
/10.23.71.2 with status NORMAL - alive false

In extreme cases, nodes didn't even come up again after a restart with an error that there
are no seed hosts (sorry, don't have the error message in current logs), but the seed host(s)
were definitively up and running. A reboot fixed that issue, starting the node again and again
did not help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message