incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasha Dolgy <sdo...@gmail.com>
Subject Re: Removing a node ...
Date Wed, 09 Mar 2011 20:00:26 GMT
Hi,

Checked logs on node where decommission command was performed and on
other nodes, and no error messages.  Just info messages.  Although the
behaviour and circumstances are exact, I'm wondering if
https://issues.apache.org/jira/browse/CASSANDRA-2072 has something to
do with it.  Again, this is with 0.7.0 ... Some more details on what i
did:

$CASSANDRA_HOME/bin/nodetool -h 10.0.0.1 decommission

When I view the ring from another node:


10.0.0.1    Down   Leaving 218.71 KB       21.76%
61078635599166706937511052402724559481

I see this message.  Great ... but after an hour of waiting, I give up
and try to force the removal of the token:

nodetool -h 10.0.0.2 removetoken 61078635599166706937511052402724559481
Exception in thread "main" java.lang.UnsupportedOperationException:
Node /10.0.0.1 is already being removed.

Ok then... this is interesting:

nodetool -h 10.0.0.2 removetoken status
RemovalStatus: No token removals in process.

I don't get it.  How do I gracefully remove a node?  Finally, I killed
the node on 10.0.0.1 and removed it's data.  Ungraceful.  I then went
to the other nodes, still couldn't force it's removal.  Started the
node back up on 10.0.0.1 and it's rejoined the cluster ... with data
spread evenly around.  Not exactly what I wanted ... oh well....

I'm sure I've missed a concept.  So, now that I have a 3 node cluster
working and balanced, I turn off cassandra on 10.0.0.1 and check the
ring from another node:

nodetool -h 10.0.0.2 ring

10.0.0.3  Up     Normal  224.21 KB       40.78%
24053088190195663439419935163232881936
10.0.0.1    Down   Normal  213.51 KB       36.78%
86624712919272143003828971968762407027
10.0.0.2    Up     Normal  244.42 KB       22.44%
124804735337540159479107746638263794797

Now, to try and remove that node by removing the token:

nodetool -h 10.0.0.1 removetoken 86624712919272143003828971968762407027

Job done, the node is gone...

nodetool -h 10.0.0.2 ring
10.0.0.3  Up     Normal  224.21 KB       40.78%
24053088190195663439419935163232881936
10.0.0.2    Up     Normal  244.42 KB       59.22%
124804735337540159479107746638263794797

-sd


On Wed, Mar 9, 2011 at 8:43 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>
> Did you check log for errors?
>
> (ASF mailing lists prefer plain text; html has a higher spam score.)
>
> On Wed, Mar 9, 2011 at 11:59 AM, Sasha Dolgy <sdolgy@gmail.com> wrote:
> > mail servers keep catching up in spam filters.  Something about being a
> > loyal gmail user ...!
> > Let's try this again:
> >
> > further to this ...using cassandra 0.7.0

Mime
View raw message