cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From B R <software.research.w...@gmail.com>
Subject Re: Issue with nodetool clearsnapshot
Date Wed, 07 Mar 2012 05:39:33 GMT
Thanks a lot, Aaron. Our cluster is much stable now. We'll look at
upgrading to 1.x in the coming weeks.

On Tue, Mar 6, 2012 at 2:33 PM, aaron morton <aaron@thelastpickle.com>wrote:

> 1)Since you mentioned hard links, I would like to add that our data
> directory itself is a sym-link. Could that be causing an issue ?
>
> Seems unlikely.
>
> I restarted the node and it went about deleting the files and the disk
> space has been released. Can this be done using nodetool, and without
> restarting ?
>
> Under 0.8.x they are deleted when the files are no longer in use and when
> JVM GC free's all references. You can provoke this by getting the JVM to GC
> using JConsole or another JMX client.
>
> If there is not enough disk free space GC is forced and free space
> reclaimed.
>
> Under 1.x file handles are counted and the files are quickly deleted.
>
> Cheers
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6/03/2012, at 7:38 AM, B R wrote:
>
> Hi Aaron,
>
> 1)Since you mentioned hard links, I would like to add that our data
> directory itself is a sym-link. Could that be causing an issue ?
>
> 2)Yes, there are 0 byte files of the same numbers
> in Keyspace1 directory
> 0 Mar  4 01:33 Standard1-g-7317-Compacted
> 0 Mar  3 22:58 Standard1-g-7968-Compacted
> 0 Mar  3 23:10 Standard1-g-8778-Compacted
> 0 Mar  3 23:47 Standard1-g-8782-Compacted
> ...
>
> I restarted the node and it went about deleting the files and the disk
> space has been released. Can this be done using nodetool, and without
> restarting ?
>
> Thanks.
>
> On Mon, Mar 5, 2012 at 10:59 PM, aaron morton <aaron@thelastpickle.com>wrote:
>
>>   It seems that instead of removing the snapshot, clearsnapshot moved
>> the data files from the snapshot directory to the parent directory and the
>> size of the data for that keyspace has doubled.
>>
>> That is not possible, there is only code there to delete a files in the
>> snapshot.
>>
>> Note that in the snapshot are hard links to the files in the data dir.
>> Deleting / clearing the snapshot will not delete the files from the data
>> dir if they are still in use.
>>
>>  Many of the files are looking like duplicates.
>>
>> in Keyspace1 directory
>> 156987786084 Jan 21 03:18 Standard1-g-7317-Data.db
>> 156987786084 Mar  4 01:33 Standard1-g-8850-Data.db
>>
>> Under 0.8.x files are not immediately deleted. Did the data directory
>> contain zero size -Compacted files with the same number ?
>>
>> Cheers
>>
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 5/03/2012, at 11:50 PM, B R wrote:
>>
>> Version 0.8.9
>>
>> We run a 2 node cluster with RF=2. We ran a scrub and after that ran the
>> clearsnapshot to remove the backup snapshot created by scrub. It seems that
>> instead of removing the snapshot, clearsnapshot moved the data files from
>> the snapshot directory to the parent directory and the size of the data for
>> that keyspace has doubled. Many of the files are looking like duplicates.
>>
>> in Keyspace1 directory
>> 156987786084 Jan 21 03:18 Standard1-g-7317-Data.db
>> 156987786084 Mar  4 01:33 Standard1-g-8850-Data.db
>> 118211555728 Jan 31 12:50 Standard1-g-7968-Data.db
>> 118211555728 Mar  3 22:58 Standard1-g-8840-Data.db
>> 116902342895 Feb 25 02:04 Standard1-g-8832-Data.db
>> 116902342895 Mar  3 22:10 Standard1-g-8836-Data.db
>> 93788425710 Feb 21 04:20 Standard1-g-8791-Data.db
>> 93788425710 Mar  4 00:29 Standard1-g-8845-Data.db
>> .....
>>
>> Even though the nodetool ring command shows the correct data size for the
>> node, the du -sh on the keyspace directory gives double the size.
>>
>> Can you guide us to proceed from this situation ?
>>
>> Thanks.
>>
>>
>>
>
>

Mime
View raw message