cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Agrawal <>
Subject RE: Tombstone gc after gc grace seconds
Date Fri, 30 Jan 2015 16:32:09 GMT
I did a small test. I wrote data to 4 different column family. 30MB of data.
256 rowkeys and 100K columns on an average.
And then deleted all data from all of them.

1.       Md_normal - created using default compaction parameters and Gc Grace seconds was
5 seconds. Data was written and then deleted. Compaction was ran using "nodetool compact keyspace
columnfamily" - I see full disk data, but cannot query columns(since data was deleted consistent
behavior) and cannot query rows in cqlsh. Hits timeout.

2.       Md_test - created using following compact parameters - "compaction={'tombstone_threshold':
'0.000001', 'class': 'SizeTieredCompactionStrategy'}" and Gc Grace seconds was 5 seconds.
Disksize is reduced, and am able to query rows which return 0.

3.       Md_test2 - created using following compact parameters - "compaction={'tombstone_threshold':
'0.0', 'class': 'SizeTieredCompactionStrategy'}". Disksize is reduced, not able to query rows
using cqlsh. Hits timeout.

4.       Md_forcecompact - created using compaction parameters "compaction={'unchecked_tombstone_compaction':
'true', 'class': 'SizeTieredCompactionStrategy'}" and Gc Grace seconds was 5 seconds. Data
was written and then deleted. I see full disk data, but cannot query any data using mddbreader
and cannot query rows in cqlsh. Hits timeout.

Next day sizes were -
30M     ./md_forcecompact
4.0K    ./md_test
304K    ./md_test2
30M     ./md_normal

Feel of the data that we have is -
8000 rowkeys per day and columns are added throughout the day. 300K columns on an average
per rowKey.

From: Alain RODRIGUEZ []
Sent: Friday, January 30, 2015 4:26 AM
Subject: Re: Tombstone gc after gc grace seconds

The point is that all the "parts" or "fragments" of the row need to be in the SSTables implied
in the compaction for C* to be able to evict the row effectively.

My understanding of those parameters is that they will trigger a compaction on the SSTable
that exceed this ratio. This will work properly if you never "update" a row (by modifying
a value or adding a column). If your workflow is something like "Write once per partition
key", this parameter will do the job.

If you have fragments, you might trigger this compaction for nothing. In the case of frequently
updated rows (like when using wide rows / time series) your only way to get rid of tombstone
is a major compaction.

That's how I understand this.

Hope this help,



2015-01-30 1:29 GMT+01:00 Mohammed Guller <<>>:
Ravi -

It may help.

What version are you running? Do you know if minor compaction is getting triggered at all?
One way to check would be see how many sstables the data directory has.


From: Ravi Agrawal [<>]
Sent: Thursday, January 29, 2015 1:29 PM
Subject: RE: Tombstone gc after gc grace seconds

I saw there are 2 more interesting parameters -

a.       tombstone_threshold - A ratio of garbage-collectable tombstones to all contained
columns, which if exceeded by the SSTable triggers compaction (with no other SSTables) for
the purpose of purging the tombstones. Default value - 0.2

b.      unchecked_tombstone_compaction - True enables more aggressive than normal tombstone
compactions. A single SSTable tombstone compaction runs without checking the likelihood of
success. Cassandra 2.0.9 and later.
Could I use these to get what I want?
Problem I am encountering is even long after gc_grace_seconds I see no reduction in disk space
until I run compaction manually. I was thinking to make tombstone threshold close to 0 and
unchecked compaction set to true.
Also we are not running nodetool repair on weekly basis as of now.

From: Eric Stevens []
Sent: Monday, January 26, 2015 12:11 PM
Subject: Re: Tombstone gc after gc grace seconds

My understanding is consistent with Alain's, there's no way to force a tombstone-only compaction,
your only option is major compaction.  If you're using size tiered, that comes with its own

I wonder if there's a technical limitation that prevents introducing a shadowed data cleanup
style operation (overwritten data, including deletes, plus tombstones past their gc grace
period); or maybe even couple it directly with cleanup since most of the work (rewriting old
SSTables) would be identical.  I can't think of something off the top of my head, but it would
be so useful that it seems like there's got to be something I'm missing.

On Mon, Jan 26, 2015 at 4:15 AM, Alain RODRIGUEZ <<>>
I don't think that such a thing exists as SSTables are immutable. You compact it entirely
or you don't. Minor compaction will eventually evict tombstones. If it is too slow, AFAIK,
the "better" solution is a major compaction.



2015-01-23 0:00 GMT+01:00 Ravi Agrawal <<>>:
I want to trigger just tombstone compaction after gc grace seconds is completed not nodetool
compact keyspace column family.
Anyway I can do that?


View raw message