Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 30 Apr 2014 20:31:16 +0000 (UTC)
From: "Richard Low (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12711569.1398889465412.209198.1398889876465@arcas>
In-Reply-To: <JIRA.12711569.1398889465412@arcas>
References: <JIRA.12711569.1398889465412@arcas>
Subject: [jira] [Comment Edited] (CASSANDRA-7122) Replacement nodes have
 null entries in system.peers
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986029#comment-13986029 ] 

Richard Low edited comment on CASSANDRA-7122 at 4/30/14 8:30 PM:
-----------------------------------------------------------------

It looks to be caused by StorageService.onChange ignoring state changes when %{code}Gossiper.instance.isDeadState(epState)%{/code}. When a replacement node comes up, it is in the hibernate state, which is counted as dead. Since the info like DC and rack never changes, it is never resent so is never persisted to system.peers.

Is it safe to special case hibernate and persist the peer info? Dead states were ignored by CASSANDRA-6053, maybe that was overly restrictive?


was (Author: rlow):
It looks to be caused by StorageService.onChange ignoring state changes when %{code}Gossiper.instance.isDeadState(epState)%{code}. When a replacement node comes up, it is in the hibernate state, which is counted as dead. Since the info like DC and rack never changes, it is never resent so is never persisted to system.peers.

Is it safe to special case hibernate and persist the peer info? Dead states were ignored by CASSANDRA-6053, maybe that was overly restrictive?

> Replacement nodes have null entries in system.peers
> ---------------------------------------------------
>
>                 Key: CASSANDRA-7122
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7122
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Richard Low
>
> If a node is replaced with -Dcassandra.replace_address, the new node has mostly null entries in system.peers:
> {code}
> > select * from system.peers;
>  peer      | data_center | host_id | rack | release_version | rpc_address | schema_version | tokens
> -----------+-------------+---------+------+-----------------+-------------+----------------+--------------------------
>  127.0.0.3 |        null |    null | null |            null |        null |           null | {'-3074457345618258602'}
> {code}
> To reproduce, simply kill a node and replace it. The entries are correctly populated if the replacement node is restarted but they are never populated if it isn't.
> I can think of at least two bad consequences of this:
> 1. Drivers like Datastax java-driver use the peers table to find the rpc_address and location info of a node. If the entires are null it assumes rpc_address=ip and the node is in the local DC.
> 2. When using GossipingPropertyFileSnitch and node won't persist the DC/rack of another node so may not be able to locate it during restarts.
> I reproduced in 1.2.15 but from inspection it looks to be present in 1.2.16 and 2.0.7.


--
This message was sent by Atlassian JIRA
(v6.2#6252)