incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jabbar Azam <aja...@gmail.com>
Subject Re: Recovering from a faulty cassandra node
Date Wed, 20 Mar 2013 12:34:02 GMT
I've added the node with a different IP address and after disabling the
firewall data is being streamed from the existing nodes to the wiped node.
I'll do a cleanup, followed by remove node once it's done.

I've also added the new node to the existing nodes'
cassandra-topology.properties file and restarted them. I also found I had
iptables switched on and couldn't understand why the wiped node couldn't
see the cluster. Not sure if I needed to change
cassandra-topology.properties file on the existing nodes.




On 19 March 2013 15:49, Jabbar Azam <ajazam@gmail.com> wrote:

> Do I use removenode before adding the reinstalled node or after?
>
>
> On 19 March 2013 15:45, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
>
>> In 1.2, you may want to use the nodetool removenode if your server i
>> broken or unreachable, else I guess nodetool decommission remains the good
>> way to remove a node. (
>> http://www.datastax.com/docs/1.2/references/nodetool)
>>
>> When this node is out, rm -rf /yourpath/cassandra/* on this serveur,
>> change the configuration if needed (not sure about the auto_bootstrap
>> param) and start Cassandra on that node again. It should join the ring as a
>> new node.
>>
>> Good luck.
>>
>>
>> 2013/3/19 Hiller, Dean <Dean.Hiller@nrel.gov>
>>
>> Since you "cleared" out that node, it IS the replacement node.
>>>
>>> Dean
>>>
>>> From: Jabbar Azam <ajazam@gmail.com<mailto:ajazam@gmail.com>>
>>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
>>> <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>>> Date: Tuesday, March 19, 2013 9:29 AM
>>> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
>>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>>> Subject: Re: Recovering from a faulty cassandra node
>>>
>>> Hello Dean.
>>>
>>> I'm using vnodes so can't specify a token. In addition I can't follow
>>> the replace node docs because I don't have a replacement node.
>>>
>>>
>>> On 19 March 2013 15:25, Hiller, Dean <Dean.Hiller@nrel.gov<mailto:
>>> Dean.Hiller@nrel.gov>> wrote:
>>> I have not done this as of yet but from all that I have read your best
>>> option is to follow the replace node documentation which I belive you need
>>> to
>>>
>>>
>>>  1.  Have the token be the same BUT add 1 to it so it doesn't think it's
>>> the same computer
>>>  2.  Have the bootstrap option set or something so streaming takes
>>> affect.
>>>
>>> I would however test that all out in QA to make sure it works and if you
>>> have QUOROM reads/writes a good part of that test would be to take node X
>>> down after your node Y is back in the cluster to make sure reads/writes are
>>> working on the node you fixed…..you just need to make sure node X shares
>>> one of the token ranges of node Y AND your writes/reads are in that token
>>> range.
>>>
>>> Dean
>>>
>>> From: Jabbar Azam <ajazam@gmail.com<mailto:ajazam@gmail.com><mailto:
>>> ajazam@gmail.com<mailto:ajazam@gmail.com>>>
>>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org
>>> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
<
>>> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
>>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
>>> Date: Tuesday, March 19, 2013 8:51 AM
>>> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
>>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" <
>>> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
>>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
>>> Subject: Recovering from a faulty cassandra node
>>>
>>> Hello,
>>>
>>> I am using Cassandra 1.2.2 on a 4 node test cluster with vnodes. I
>>> waited for over a week to insert lots of data into the cluster. During the
>>> end of the process one of the nodes had a hardware fault.
>>>
>>> I have fixed the hardware fault but the filing system on that node is
>>> corrupt so I'll have to reinstall the OS and cassandra.
>>>
>>> I can think of two ways of reintegrating the host into the cluster
>>>
>>> 1) shrink the cluster to three nodes and add the node into the cluster
>>>
>>> 2) Add the node into the cluster without shrinking
>>>
>>> I'm not sure of the best approach to take and I'm not sure how to
>>> achieve each step.
>>>
>>> Can anybody help?
>>>
>>>
>>> --
>>> Thanks
>>>
>>>  Jabbar Azam
>>>
>>>
>>>
>>> --
>>> Thanks
>>>
>>> Jabbar Azam
>>>
>>
>>
>
>
> --
> Thanks
>
> Jabbar Azam
>



-- 
Thanks

Jabbar Azam

Mime
View raw message