incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Curious Patient <mongoma...@gmail.com>
Subject problem removing dead node from ring
Date Tue, 03 Jun 2014 15:41:27 GMT
One of the nodes in a cassandra cluster has died.

I'm using cassandra 2.0.7 throughout.

When I do a nodetool status this is what I see (real addresses have been
replaced with fake 10 nets)

    [root@beta-new:/opt] #nodetool status
    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address         Load       Tokens  Owns   Host ID
            Rack
    UN  10.10.1.94  171.02 KB  256     49.4%
 fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
    DN  10.10.1.98     ?          256     50.6%
 f2a48fc7-a362-43f5-9061-4bb3739fdeaf  rack1

I tried to get the token ID for the down node by doing a nodetool ring
command, grepping for the IP and doing a head -1 to get the initial one.

    [root@beta-new:/opt] #nodetool ring | grep 10.10.1.98 | head -1
    10.10.1.98     rack1       Down   Normal  ?               50.59%
       -9042969066862165996

I then started following this documentation on how to replace the node:

[
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html?scroll=task_ds_aks_15q_gk][1
]

So I installed cassandra on a new node but did not start it.

Set the following options:

    cluster_name: 'Test Cluster'
    seed_provider:
          - seeds: "10.10.1.94"
    listen_address: 10.10.1.94
    endpoint_snitch: SimpleSnitch

And set the initial token of the new install as the token -1 of the node
I'm trying to replace in cssandra.yaml:

    initial_token: -9042969066862165995

And after making sure there was no data yet in:
      /var/lib/cassandra

I started up the database:

    [root@web2:/etc/alternatives/cassandrahome] #./bin/cassandra -f
-Dcassandra.replace_address=10.10.1.98


The documentation I link to above says to use the replace_address directive
on the command line rather than cassandra-env.sh if you have a tarball
install (which we do) as opposed to a package install.

After I start it up, cassandra fails with the following message:

    Exception encountered during startup: Cannot replace_address /
10.10.10.98 because it doesn't exist in gossip



So I'm wondering at this point if I've missed any steps or if there is
anything else I can try to replace this dead cassandra node?

Mime
View raw message