cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Or Sher <or.sh...@gmail.com>
Subject Re: Replacing nodes disks
Date Sun, 21 Dec 2014 09:20:42 GMT
What I want to do is kind of replacing a dead node -
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
But replacing it with a clean node with the same IP and hostname.

On Sun, Dec 21, 2014 at 9:53 AM, Or Sher <or.sher1@gmail.com> wrote:

> Thanks guys.
> I have to replace all data disks, so I don't have another large enough
> local disk to move the data to.
> If I'll have no choice, I will backup the data before on some other node
> or something, but I'd like to avoid it.
> I would really love letting Cassandra do it thing and rebuild itself.
> Did anybody handled such cases that way (Letting Cassandra rebuild it's
> data?)
> Although there are no documented procedure for it, It should be possible
> right?
>
> On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten <j.kesten@enercast.de> wrote:
>
>> Hi Or,
>>
>> I did some sort of this a while ago. If your machines do have a free disk
>> slot - just put another disk there and use it as another
>> data_file_directory.
>>
>> If not - as in my case:
>>
>> - grab an usb dock for disks
>> - put the new one in there, plug in, format, mount to /mnt etc.
>> - I did an online rsync from /var/lib/cassandra/data to /mnt
>> - after that, bring cassandra down
>> - do another rsync from /var/lib/cassandra/data to /mnt (should be
>> faster, as sstables do not change, minimizes downtime)
>> - if you need adjust /etc/fstab if needed
>> - shutdown the node
>> - swap disks
>> - power on the node
>> - everything should be fine ;-)
>>
>> Of course you will need a replication factor > 1 for this to work ;-)
>>
>> Just my 2 cents,
>> Jan
>>
>> rsync the full contents there,
>>
>> Am 18.12.2014 um 16:17 schrieb Or Sher:
>>
>>  Hi all,
>>>
>>> We have a situation where some of our nodes have smaller disks and we
>>> would like to align all nodes by replacing the smaller disks to bigger ones
>>> without replacing nodes.
>>> We don't have enough space to put data on / disk and copy it back to the
>>> bigger disks so we would like to rebuild the nodes data from other replicas.
>>>
>>> What do you think should be the procedure here?
>>>
>>> I'm guessing it should be something like this but I'm pretty sure it's
>>> not enough.
>>> 1. shutdown C* node and server.
>>> 2. replace disks + create the same vg lv etc.
>>> 3. start C* (Normally?)
>>> 4. nodetool repair/rebuild?
>>> *I think I might get some consistency issues for use cases relying on
>>> Quorum reads and writes for strong consistency.
>>> What do you say?
>>>
>>> Another question is (and I know it depends on many factors but I'd like
>>> to hear an experienced estimation): How much time would take to rebuild a
>>> 250G data node?
>>>
>>> Thanks in advance,
>>> Or.
>>>
>>> --
>>> Or Sher
>>>
>>
>>
>
>
> --
> Or Sher
>



-- 
Or Sher

Mime
View raw message