incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Does a scrub remove deleted/expired columns?
Date Sun, 16 Dec 2012 20:17:56 GMT
> periodically trimming the row by by deleting the oldest columns, the deleted columns won't
get cleaned up until all fragments of the row exist in a single sstable and that sstable undergoes
a compaction?
Nope. 
They are purged when all of the fragments of the row exist in the same SSTabels (plural) being
compacted. 

Say you create a row and write to it for a while, it may be spread into 2 or 3 new stables.
When there are 4 they are compacted into one, which will be bigger than the original 4. When
there are 4 at the next size bucket they are compacted and so on. 

If you row exists in one size bucket only it GC will be purged. 

If you have a row you have been writing to for a long time it may be spread out in many buckets.
That's not normally a big problem, but if you also do lots of deletes the tombstones will
not get purged. 

Hope that helps. 

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 14/12/2012, at 4:45 PM, Mike Smith <mike@mailchannels.com> wrote:

> Thanks for the great explanation.
> 
> I'd just like some clarification on the last point. Is it the case that if I constantly
add new columns to a row, while periodically trimming the row by by deleting the oldest columns,
the deleted columns won't get cleaned up until all fragments of the row exist in a single
sstable and that sstable undergoes a compaction?
> 
> If my understanding is correct, do you know if 1.2 will enable cleanup of columns in
rows that have scattered fragments? Or, should I take a different approach?
> 
> 
> 
> On Thu, Dec 13, 2012 at 5:52 PM, aaron morton <aaron@thelastpickle.com> wrote:
>>  Is it possible to use scrub to accelerate the clean up of expired/deleted data?
> No.
> Scrub, and upgradesstables, are used to re-write each file on disk. Scrub may remove
some rows from a file because of corruption, however upgradesstables will not. 
> 
> If you have long lived rows and a mixed work load of writes and deletes there are a couple
of options. 
> 
> You can try levelled compaction http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
> 
> You can tune the default sized tiered compaction by increasing the min_compaction_threshold.
This will increase the number of files that must exist in each size tier before it will be
compacted. As a result the speed at which rows move into the higher tiers will slow down.

> 
> Note that having lots of files may have a negative impact on read performance. You can
measure this my looking at the SSTables per read metric in the cfhistograms. 
> 
> Lastly you can run a user defined or major compaction. User defined compaction is available
via JMX and allows you to compact any file you want. Manual / major compaction is available
via node tool. We usually discourage it's use as it will create one big file that will not
get compacted for a while. 
> 
> 
> For background the tombstones / expired columns for a row are only purged from the database
when all fragments of the row are  in the files been compacted. So if you have an old row
that is spread out over many files it may not get purged. 
> 
> Hope that helps. 
> 
> 
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 14/12/2012, at 3:01 AM, Mike Smith <mike@mailchannels.com> wrote:
> 
>> I'm using 1.0.12 and I find that large sstables tend to get compacted infrequently.
I've got data that gets deleted or expired frequently. Is it possible to use scrub to accelerate
the clean up of expired/deleted data?
>> 
>> -- 
>> Mike Smith
>> Director Development, MailChannels
>> 
> 
> 
> 
> 
> -- 
> Mike Smith
> Director Development, MailChannels
> 


Mime
View raw message