Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 8 Aug 2016 08:06:20 +0000 (UTC)
From: "Benjamin Roth (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12995561.1470641682000.246814.1470643580480@Atlassian.JIRA>
In-Reply-To: <JIRA.12995561.1470641682000@Atlassian.JIRA>
References: <JIRA.12995561.1470641682000@Atlassian.JIRA> <JIRA.12995561.1470641682567@arcas>
Subject: [jira] [Commented] (CASSANDRA-12405) node health status
 inconsistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 08 Aug 2016 08:06:22 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411451#comment-15411451 ] 

Benjamin Roth commented on CASSANDRA-12405:
-------------------------------------------

I recognized, that the node "cas1" hat different seeds configured. Seems that puppet did not run. Don't know if that relates to the issue.

Seeds are: 10.23.71.101,10.23.71.1,10.23.71.2
On cas1 I think they where 10.23.71.2,10.23.71.3

> node health status inconsistent
> -------------------------------
>
>                 Key: CASSANDRA-12405
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12405
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 3.9, Linux Xenial
>            Reporter: Benjamin Roth
>
> ATM we run a 4 node cluster with cassandra 3.9
> Due to another issue (hanging repairs) I am forced to restart nodes from time to time. Before I restart a node, all nodes are listed as UP from any other node.
> When I restart one node in the cluster, the health statuses of other nodes are affected as well.
> After having restarted node "cas", the "nodetool status" output on all nodes looks like this during startup phase of cas1: https://gist.github.com/brstgt/9be77470814d2fd160617a1c06579804
> After cas1 is up again, I restart cas2. During the startup phase of cas2 the status looks like this:
> https://gist.github.com/brstgt/d27ef540b2389b3a7d2d015ab83af547
> The nodetool output goes along with log messages like this:
> 2016-08-08T07:30:06+00:00 cas1 [GossipTasks: 1] org.apache.cassandra.gms.Gossiper Convicting /10.23.71.3 with status NORMAL - alive false
> 2016-08-08T07:30:06+00:00 cas1 [GossipTasks: 1] org.apache.cassandra.gms.Gossiper Convicting /10.23.71.2 with status NORMAL - alive false
> In extreme cases, nodes didn't even come up again after a restart with an error that there are no seed hosts (sorry, don't have the error message in current logs), but the seed host(s) were definitively up and running. A reboot fixed that issue, starting the node again and again did not help.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)