incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: replace dead node? " token -1 "
Date Wed, 15 Aug 2012 09:00:55 GMT
considering there is this minor "old node hanging around" issue, would the
old T-1 approach sound more attractive?
that way you don't necessarily have to remove the dead token immediately,
but could come back the next day, or even a week  later. T-1 would behave
essentially the same in terms of partitioning the data range.


Thanks
Yang

On Wed, Aug 15, 2012 at 1:39 AM, Yang <teddyyyy123@gmail.com> wrote:

> ok,  I see, the cassandra.replace_token  setting essentially  executes the
> manual removeToken step. so the dead node should be removed.
>
> is this the "old node hanging around" issue that you described?
> https://issues.apache.org/jira/browse/CASSANDRA-3259
> looks this JIRA is fixed in 1.0x already, so it's another issue?
>
> Thanks
> Yang
>
> On Tue, Aug 14, 2012 at 11:03 PM, Yang <teddyyyy123@gmail.com> wrote:
>
>> Jim:
>>
>> thanks a  lot for the info.
>>
>> when you say "old nodes sometimes hanging around as "unreachable nodes"
>> when describing cluster", you mean after the new node boots up and assumes
>> ownership of the same token, you have not manually run nodetool
>> removeToken, right? this kind of makes sense --- since it seems that the
>> membership being gossiped around still contains the dead node (which is
>> represented by a different AWS internal ip), though the same token is being
>> associated to both dead and new nodes ??? I'm getting a bit confused
>> here....
>>
>>
>> I think previously when I boot up a new node with the same token, while
>> the old host is dead, the other nodes on the
>> ring says something like "this token xxxxxx is already owned by
>> old_node_ip_here,...... ".  I don't remember exactly the behavior now,
>> that's why I'm cautious of using T instead of T-1.
>>
>>
>> I'm doing more tests to confirm this behavior
>>
>> Thanks
>> Yang
>>
>>
>> On Tue, Aug 14, 2012 at 10:17 PM, Jim Cistaro <jcistaro@netflix.com>wrote:
>>
>>>  We use priam to replace nodes using replace_token.  We do see some
>>> issues (currently on 1.0.9, as well as earlier versions) with replace_token.
>>>
>>>  Apparently there are some known issues with replace_token.  We have
>>> experienced the old nodes sometimes hanging around as "unreachable nodes"
>>> when describing cluster.  Also, we have experienced problems where moving
>>> the new node causes the old "replaced" node to resurrect for the token that
>>> was outgoing during the move.
>>>
>>>  You can notice these old nodes hanging around in logs.  You will see
>>> messages like:
>>> StorageService.java (line 1020) Nodes /<old_ip> and /<new_ip> have
the
>>> same token NNNNNNNNNN.  Ignoring /<old_ip>.
>>>
>>>  We have then had to "nt removetoken" to clean things up after moves.
>>>  We are also investigating using method unsafeAssassinateEndpoint (via jmx)
>>> to clean up some of the unreachables.
>>>
>>>  Like I said, we still use replace_token, but be aware of these
>>> possible inconveniences.
>>>
>>>  Jim Cistaro
>>> Netflix Cassandra Operations
>>>
>>>
>>>   From: Yang <teddyyyy123@gmail.com>
>>> Reply-To: <user@cassandra.apache.org>
>>> Date: Tue, 14 Aug 2012 21:58:30 -0700
>>> To: <user@cassandra.apache.org>
>>> Subject: Re: replace dead node? " token -1 "
>>>
>>>  thanks Aaron, it has been a while since i last checked the code,  I'll
>>> read it to understand it more
>>>  On Aug 14, 2012 8:48 PM, "aaron morton" <aaron@thelastpickle.com>
>>> wrote:
>>>
>>>>  Using this method, when choosing the new <Token>, should we still
use
>>>> the T-1 ?
>>>>
>>>> (AFAIK) No.
>>>> replace_token is used when you want to replace a node that is dead. In
>>>> this case the dead node will be identified by its token.
>>>>
>>>>  if so, would the duplicate token (same token but different ip) cause
>>>> problems?
>>>>
>>>>  If the nodes are bootstrapping an error is raised.
>>>> Otherwise the token ownership is passed to the new node.
>>>>
>>>>  Cheers
>>>>
>>>>    -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>>  On 15/08/2012, at 11:07 AM, Yang <teddyyyy123@gmail.com> wrote:
>>>>
>>>>  previously when a node dies, I remember the documents describes that
>>>> it's better to assign T-1 to the new node,
>>>> where T was the token of the dead node.
>>>>
>>>>
>>>>  the new doc for 1.x here
>>>>
>>>>  http://wiki.apache.org/cassandra/Operations#Replacing_a_Dead_Node
>>>>
>>>>
>>>>  shows a new way to  pass in cassandra.replace_token=<Token>
>>>> for the new node.
>>>> Using this method, when choosing the new <Token>, should we still use
>>>> the T-1 ?
>>>>
>>>>
>>>>  Also in Priam code:
>>>>
>>>> https://github.com/Netflix/Priam/blob/master/priam/src/main/java/com/netflix/priam/identity/InstanceIdentity.java
>>>>
>>>>  line 148, it does not seem that Priam does the "-1" thing, but
>>>> assigns the original token T to the new node.
>>>> if so, would the duplicate token (same token but different ip) cause
>>>> problems?
>>>>
>>>>
>>>>  Thanks
>>>> Yang
>>>>
>>>>
>>>>
>>
>

Mime
View raw message