Thanks a lot, Aaron. Our cluster is much stable now. We'll look at upgrading to 1.x in the coming weeks.
1)Since you mentioned hard links, I would like to add that our data directory itself is a sym-link. Could that be causing an issue ?Seems unlikely.I restarted the node and it went about deleting the files and the disk space has been released. Can this be done using nodetool, and without restarting ?Under 0.8.x they are deleted when the files are no longer in use and when JVM GC free's all references. You can provoke this by getting the JVM to GC using JConsole or another JMX client.If there is not enough disk free space GC is forced and free space reclaimed.Under 1.x file handles are counted and the files are quickly deleted.CheersOn 6/03/2012, at 7:38 AM, B R wrote:Hi Aaron,1)Since you mentioned hard links, I would like to add that our data directory itself is a sym-link. Could that be causing an issue ?2)Yes, there are 0 byte files of the same numbersin Keyspace1 directory0 Mar 4 01:33 Standard1-g-7317-Compacted0 Mar 3 22:58 Standard1-g-7968-Compacted0 Mar 3 23:10 Standard1-g-8778-Compacted0 Mar 3 23:47 Standard1-g-8782-Compacted...I restarted the node and it went about deleting the files and the disk space has been released. Can this be done using nodetool, and without restarting ?Thanks.
On Mon, Mar 5, 2012 at 10:59 PM, aaron morton <email@example.com> wrote:On 5/03/2012, at 11:50 PM, B R wrote:Version 0.8.9
We run a 2 node cluster with RF=2. We ran a scrub and after that ran the clearsnapshot to remove the backup snapshot created by scrub. It seems that instead of removing the snapshot, clearsnapshot moved the data files from the snapshot directory to the parent directory and the size of the data for that keyspace has doubled. Many of the files are looking like duplicates.
in Keyspace1 directory
156987786084 Jan 21 03:18 Standard1-g-7317-Data.db
156987786084 Mar 4 01:33 Standard1-g-8850-Data.db
118211555728 Jan 31 12:50 Standard1-g-7968-Data.db
118211555728 Mar 3 22:58 Standard1-g-8840-Data.db
116902342895 Feb 25 02:04 Standard1-g-8832-Data.db
116902342895 Mar 3 22:10 Standard1-g-8836-Data.db
93788425710 Feb 21 04:20 Standard1-g-8791-Data.db
93788425710 Mar 4 00:29 Standard1-g-8845-Data.db
Even though the nodetool ring command shows the correct data size for the node, the du -sh on the keyspace directory gives double the size.
Can you guide us to proceed from this situation ?