incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Cistaro <jcist...@netflix.com>
Subject Re: replace dead node? " token -1 "
Date Wed, 15 Aug 2012 05:17:43 GMT
We use priam to replace nodes using replace_token.  We do see some issues (currently on 1.0.9,
as well as earlier versions) with replace_token.

Apparently there are some known issues with replace_token.  We have experienced the old nodes
sometimes hanging around as "unreachable nodes" when describing cluster.  Also, we have experienced
problems where moving the new node causes the old "replaced" node to resurrect for the token
that was outgoing during the move.

You can notice these old nodes hanging around in logs.  You will see messages like:
StorageService.java (line 1020) Nodes /<old_ip> and /<new_ip> have the same token
NNNNNNNNNN.  Ignoring /<old_ip>.

We have then had to "nt removetoken" to clean things up after moves.  We are also investigating
using method unsafeAssassinateEndpoint (via jmx) to clean up some of the unreachables.

Like I said, we still use replace_token, but be aware of these possible inconveniences.

Jim Cistaro
Netflix Cassandra Operations


From: Yang <teddyyyy123@gmail.com<mailto:teddyyyy123@gmail.com>>
Reply-To: <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Tue, 14 Aug 2012 21:58:30 -0700
To: <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: replace dead node? " token -1 "


thanks Aaron, it has been a while since i last checked the code,  I'll read it to understand
it more

On Aug 14, 2012 8:48 PM, "aaron morton" <aaron@thelastpickle.com<mailto:aaron@thelastpickle.com>>
wrote:
Using this method, when choosing the new <Token>, should we still use the T-1 ?
(AFAIK) No.
replace_token is used when you want to replace a node that is dead. In this case the dead
node will be identified by its token.

if so, would the duplicate token (same token but different ip) cause problems?
If the nodes are bootstrapping an error is raised.
Otherwise the token ownership is passed to the new node.

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 15/08/2012, at 11:07 AM, Yang <teddyyyy123@gmail.com<mailto:teddyyyy123@gmail.com>>
wrote:

previously when a node dies, I remember the documents describes that it's better to assign
T-1 to the new node,
where T was the token of the dead node.


the new doc for 1.x here

http://wiki.apache.org/cassandra/Operations#Replacing_a_Dead_Node


shows a new way to  pass in cassandra.replace_token=<Token>
for the new node.
Using this method, when choosing the new <Token>, should we still use the T-1 ?


Also in Priam code:
https://github.com/Netflix/Priam/blob/master/priam/src/main/java/com/netflix/priam/identity/InstanceIdentity.java

line 148, it does not seem that Priam does the "-1" thing, but assigns the original token
T to the new node.
if so, would the duplicate token (same token but different ip) cause problems?


Thanks
Yang


Mime
View raw message