cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Subject Re: Drop TTLd rows: upgradesstables -a or scrub?
Date Mon, 10 Sep 2018 19:09:18 GMT
On Mon, 10 Sep 2018, 19:40 Jeff Jirsa, <jjirsa@gmail.com> wrote:

> I think it's important to describe exactly what's going on for people who
> just read the list but who don't have context. This blog does a really good
> job:
> http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
> , but briefly:
>
> - When a TTL expires, we treat it as a tombstone, because it may have been
> written ON TOP of another piece of live data, so we need to get that
> deletion marker to all hosts, just like a manual explicit delete
> - Tombstones in sstable A may shadow data in sstable B, so doing anything
> on just one sstable MAY NOT remove the tombstone - we can't get rid of the
> tombstone if sstable A overlaps another sstable with the same partition
> (which we identify via bloom filter) that has any data with a lower
> timestamp (we don't check the sstable for a shadowed value, we just look at
> the minimum live timestamp of the table)
>
> "nodetool garbagecollect" looks for sstables that overlap (partition keys)
> and combine them together, which makes tombstones past GCGS purgable and
> should remove them (and data shadowed by them).
>
> If you're on a version without nodetool garbagecollection, you can
> approximate it using user defined compaction (
> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html ) -
> it's a JMX endpoint that let's you tell cassandra to compact one or more
> sstables together based on parameters you choose. This is somewhat like
> upgradesstables or scrub, but you can combine sstables as well. If you
> choose candidates intelligently (notably, oldest sstables first, or
> sstables you know overlap), you can likely manually clean things up pretty
> quickly. At one point, I had a jar that would do single sstable at a time,
> oldest sstable first, and it pretty much worked for this purpose most of
> the time.
>
> If you have room, a "nodetool compact" on stcs will also work, but it'll
> give you one huge sstable, which will be unfortunate long term (probably
> less of a problem if you're no longer writing to this table).
>

That's a really nice refresher, thanks Jeff!

>From the nature of the data at hand and because of the SizeTiered
compaction, I would expect that more or less all tables do overlap with
each other.

Even if we would be able to identify the overlapping ones (how?), I expect
that we would have to do an equivalent of the major compaction, but (maybe)
in multiple stages. Not sure that's really worth the trouble for us.

Thanks,
--
Alex

On Mon, Sep 10, 2018 at 10:29 AM Charulata Sharma (charshar)
> <charshar@cisco.com.invalid> wrote:
>
>> Scrub takes a very long time and does not remove the tombstones. You
>> should do garbage cleaning. It immediately removes the tombstones.
>>
>>
>>
>> Thaks,
>>
>> Charu
>>
>>
>>
>> *From: *Oleksandr Shulgin <oleksandr.shulgin@zalando.de>
>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Date: *Monday, September 10, 2018 at 6:53 AM
>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Subject: *Drop TTLd rows: upgradesstables -a or scrub?
>>
>>
>>
>> Hello,
>>
>>
>>
>> We have some tables with significant amount of TTLd rows that have
>> expired by now (and more gc_grace_seconds have passed since the TTL).  We
>> have stopped writing more data to these tables quite a while ago, so
>> background compaction isn't running.  The compaction strategy is the
>> default SizeTiered one.
>>
>>
>>
>> Now we would like to get rid of all the droppable tombstones in these
>> tables.  What would be the approach that puts the least stress on the
>> cluster?
>>
>>
>>
>> We've considered a few, but the most promising ones seem to be these two:
>> `nodetool scrub` or `nodetool upgradesstables -a`.  We are using Cassandra
>> version 3.0.
>>
>>
>>
>> Now, this docs page recommends to use upgradesstables wherever possible:
>> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsScrub.html
>>
>> What is the reason behind it?
>>
>>
>>
>> From source code I can see that Scrubber the class which is going to drop
>> the tombstones (and report the total number in the logs):
>> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/Scrubber.java#L308
>>
>>
>>
>> I couldn't find similar handling in the upgradesstables code path.  Is
>> the assumption correct that this one will not drop the tombstone as a side
>> effect of rewriting the files?
>>
>>
>>
>> Any drawbacks of using scrub for this task?
>>
>>
>>
>> Thanks,
>> --
>>
>> Oleksandr "Alex" Shulgin | Senior Software Engineer | Team Flux | Data
>> Services | Zalando SE | Tel: +49 176 127-59-707
>>
>>
>>
>

Mime
View raw message