incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Removing a node ...
Date Wed, 09 Mar 2011 20:42:24 GMT
I think 2072 is something different but you should definitely upgrade
before troubleshooting further.

On Wed, Mar 9, 2011 at 2:00 PM, Sasha Dolgy <sdolgy@gmail.com> wrote:
> Hi,
>
> Checked logs on node where decommission command was performed and on
> other nodes, and no error messages.  Just info messages.  Although the
> behaviour and circumstances are exact, I'm wondering if
> https://issues.apache.org/jira/browse/CASSANDRA-2072 has something to
> do with it.  Again, this is with 0.7.0 ... Some more details on what i
> did:
>
> $CASSANDRA_HOME/bin/nodetool -h 10.0.0.1 decommission
>
> When I view the ring from another node:
>
>
> 10.0.0.1    Down   Leaving 218.71 KB       21.76%
> 61078635599166706937511052402724559481
>
> I see this message.  Great ... but after an hour of waiting, I give up
> and try to force the removal of the token:
>
> nodetool -h 10.0.0.2 removetoken 61078635599166706937511052402724559481
> Exception in thread "main" java.lang.UnsupportedOperationException:
> Node /10.0.0.1 is already being removed.
>
> Ok then... this is interesting:
>
> nodetool -h 10.0.0.2 removetoken status
> RemovalStatus: No token removals in process.
>
> I don't get it.  How do I gracefully remove a node?  Finally, I killed
> the node on 10.0.0.1 and removed it's data.  Ungraceful.  I then went
> to the other nodes, still couldn't force it's removal.  Started the
> node back up on 10.0.0.1 and it's rejoined the cluster ... with data
> spread evenly around.  Not exactly what I wanted ... oh well....
>
> I'm sure I've missed a concept.  So, now that I have a 3 node cluster
> working and balanced, I turn off cassandra on 10.0.0.1 and check the
> ring from another node:
>
> nodetool -h 10.0.0.2 ring
>
> 10.0.0.3  Up     Normal  224.21 KB       40.78%
> 24053088190195663439419935163232881936
> 10.0.0.1    Down   Normal  213.51 KB       36.78%
> 86624712919272143003828971968762407027
> 10.0.0.2    Up     Normal  244.42 KB       22.44%
> 124804735337540159479107746638263794797
>
> Now, to try and remove that node by removing the token:
>
> nodetool -h 10.0.0.1 removetoken 86624712919272143003828971968762407027
>
> Job done, the node is gone...
>
> nodetool -h 10.0.0.2 ring
> 10.0.0.3  Up     Normal  224.21 KB       40.78%
> 24053088190195663439419935163232881936
> 10.0.0.2    Up     Normal  244.42 KB       59.22%
> 124804735337540159479107746638263794797
>
> -sd
>
>
> On Wed, Mar 9, 2011 at 8:43 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Did you check log for errors?
>>
>> (ASF mailing lists prefer plain text; html has a higher spam score.)
>>
>> On Wed, Mar 9, 2011 at 11:59 AM, Sasha Dolgy <sdolgy@gmail.com> wrote:
>> > mail servers keep catching up in spam filters.  Something about being a
>> > loyal gmail user ...!
>> > Let's try this again:
>> >
>> > further to this ...using cassandra 0.7.0
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message