cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Curious Patient <mongoma...@gmail.com>
Subject Re: problem removing dead node from ring
Date Sun, 08 Jun 2014 01:18:26 GMT
Hey all,

 OK I gave removing the downed node from the cassandra ring another try.

To recap what's going on, this is what my ring looks like with nodetool
status:

[root@beta-new:~] #nodetool status

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address         Load       Tokens  Owns   Host ID
        Rack

UN  10.10.1.94  178.38 KB  256     49.4%
fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1

DN  10.10.1.98     ?          256     50.6%
f2a48fc7-a362-43f5-9061-4bb3739fdeaf  rack1

So I followed the steps in this document one more time:

http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html

And setup the following in the cassandra.yaml according to the above
instructions:

cluster_name: ‘Test Cluster'

num_tokens: 256

seed_provider:

listen_address: 10.10.1.153

auto_bootstrap: yes

broadcast_address: 10.10.1.153

endpoint_snitch: SimpleSnitch

initial_token: -9173731940639284976


The initial_token is the one belonging to the dead node that I'm trying to
get rid of.

I then make sure that the /var/lib/casssandra directory is completely empty
and run this startup command:

[root@cassandra1 cassandrahome]# ./bin/cassandra
-Dcassandra.replace_address=10.10.1.98 -f

Using the IP of the node I want to remove as the value to
casandra_replace_address

And when I do this is the error I get:

java.lang.RuntimeException: Cannot replace_address /10.10.1.98 because it
doesn't exist in gossip


So how can I get cassandra to realize that this node needs to be replaced
and that it SHOULDN'T exist in gossip because the node is down? That would
seem obvious to me, so why isn't it obvious to her? :)


Thanks

Tim









On Wed, Jun 4, 2014 at 4:36 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Tue, Jun 3, 2014 at 9:03 PM, Matthew Allen <matthew.j.allen@gmail.com>
> wrote:
>
>> Thanks Robert, this makes perfect sense.  Do you know if CASSANDRA-6961
>> will be ported to 1.2.x ?
>>
>
> I just asked driftx, he said "not gonna happen."
>
>
>> And apologies if these appear to be dumb questions, but is a repair more
>> suitable than a rebuild because the rebuild only contacts 1 replica (per
>> range), which may itself contain stale data ?
>>
>
> Exactly that.
>
> https://issues.apache.org/jira/browse/CASSANDRA-2434
>
> Discusses related issues in quite some detail. The tl;dr is that until
> 2434 is resolved, streams do not necessarily come from the node departing
> the range, and therefore the "unique replica count" is decreased by
> changing cluster topology.
>
> =Rob
>

Mime
View raw message