cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Drop TTLd rows: upgradesstables -a or scrub?
Date Mon, 10 Sep 2018 20:03:09 GMT
How much free space do you have, and how big is the table?

Switching to LCS is another option. 

-- 
Jeff Jirsa


> On Sep 10, 2018, at 12:09 PM, Oleksandr Shulgin <oleksandr.shulgin@zalando.de>
wrote:
> 
>> On Mon, 10 Sep 2018, 19:40 Jeff Jirsa, <jjirsa@gmail.com> wrote:
>> I think it's important to describe exactly what's going on for people who just read
the list but who don't have context. This blog does a really good job: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
, but briefly:
>> 
>> - When a TTL expires, we treat it as a tombstone, because it may have been written
ON TOP of another piece of live data, so we need to get that deletion marker to all hosts,
just like a manual explicit delete
>> - Tombstones in sstable A may shadow data in sstable B, so doing anything on just
one sstable MAY NOT remove the tombstone - we can't get rid of the tombstone if sstable A
overlaps another sstable with the same partition (which we identify via bloom filter) that
has any data with a lower timestamp (we don't check the sstable for a shadowed value, we just
look at the minimum live timestamp of the table)
>> 
>> "nodetool garbagecollect" looks for sstables that overlap (partition keys) and combine
them together, which makes tombstones past GCGS purgable and should remove them (and data
shadowed by them).
>> 
>> If you're on a version without nodetool garbagecollection, you can approximate it
using user defined compaction ( http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
) - it's a JMX endpoint that let's you tell cassandra to compact one or more sstables together
based on parameters you choose. This is somewhat like upgradesstables or scrub, but you can
combine sstables as well. If you choose candidates intelligently (notably, oldest sstables
first, or sstables you know overlap), you can likely manually clean things up pretty quickly.
At one point, I had a jar that would do single sstable at a time, oldest sstable first, and
it pretty much worked for this purpose most of the time. 
>> 
>> If you have room, a "nodetool compact" on stcs will also work, but it'll give you
one huge sstable, which will be unfortunate long term (probably less of a problem if you're
no longer writing to this table).
> 
> 
> That's a really nice refresher, thanks Jeff!
> 
> From the nature of the data at hand and because of the SizeTiered compaction, I would
expect that more or less all tables do overlap with each other.
> 
> Even if we would be able to identify the overlapping ones (how?), I expect that we would
have to do an equivalent of the major compaction, but (maybe) in multiple stages. Not sure
that's really worth the trouble for us.
> 
> Thanks,
> --
> Alex
> 
>>> On Mon, Sep 10, 2018 at 10:29 AM Charulata Sharma (charshar) <charshar@cisco.com.invalid>
wrote:
>>> Scrub takes a very long time and does not remove the tombstones. You should do
garbage cleaning. It immediately removes the tombstones.
>>> 
>>>  
>>> 
>>> Thaks,
>>> 
>>> Charu
>>> 
>>>  
>>> 
>>> From: Oleksandr Shulgin <oleksandr.shulgin@zalando.de>
>>> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
>>> Date: Monday, September 10, 2018 at 6:53 AM
>>> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
>>> Subject: Drop TTLd rows: upgradesstables -a or scrub?
>>> 
>>>  
>>> 
>>> Hello,
>>> 
>>>  
>>> 
>>> We have some tables with significant amount of TTLd rows that have expired by
now (and more gc_grace_seconds have passed since the TTL).  We have stopped writing more data
to these tables quite a while ago, so background compaction isn't running.  The compaction
strategy is the default SizeTiered one.
>>> 
>>>  
>>> 
>>> Now we would like to get rid of all the droppable tombstones in these tables.
 What would be the approach that puts the least stress on the cluster?
>>> 
>>>  
>>> 
>>> We've considered a few, but the most promising ones seem to be these two: `nodetool
scrub` or `nodetool upgradesstables -a`.  We are using Cassandra version 3.0.
>>> 
>>>  
>>> 
>>> Now, this docs page recommends to use upgradesstables wherever possible: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsScrub.html
>>> 
>>> What is the reason behind it?
>>> 
>>>  
>>> 
>>> From source code I can see that Scrubber the class which is going to drop the
tombstones (and report the total number in the logs): https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/Scrubber.java#L308
>>> 
>>>  
>>> 
>>> I couldn't find similar handling in the upgradesstables code path.  Is the assumption
correct that this one will not drop the tombstone as a side effect of rewriting the files?
>>> 
>>>  
>>> 
>>> Any drawbacks of using scrub for this task?
>>> 
>>>  
>>> 
>>> Thanks,
>>> --
>>> 
>>> Oleksandr "Alex" Shulgin | Senior Software Engineer | Team Flux | Data Services
| Zalando SE | Tel: +49 176 127-59-707
>>> 
>>>  

Mime
View raw message