cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Drop TTLd rows: upgradesstables -a or scrub?
Date Mon, 10 Sep 2018 17:39:44 GMT
I think it's important to describe exactly what's going on for people who
just read the list but who don't have context. This blog does a really good
job:
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
, but briefly:

- When a TTL expires, we treat it as a tombstone, because it may have been
written ON TOP of another piece of live data, so we need to get that
deletion marker to all hosts, just like a manual explicit delete
- Tombstones in sstable A may shadow data in sstable B, so doing anything
on just one sstable MAY NOT remove the tombstone - we can't get rid of the
tombstone if sstable A overlaps another sstable with the same partition
(which we identify via bloom filter) that has any data with a lower
timestamp (we don't check the sstable for a shadowed value, we just look at
the minimum live timestamp of the table)

"nodetool garbagecollect" looks for sstables that overlap (partition keys)
and combine them together, which makes tombstones past GCGS purgable and
should remove them (and data shadowed by them).

If you're on a version without nodetool garbagecollection, you can
approximate it using user defined compaction (
http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html ) -
it's a JMX endpoint that let's you tell cassandra to compact one or more
sstables together based on parameters you choose. This is somewhat like
upgradesstables or scrub, but you can combine sstables as well. If you
choose candidates intelligently (notably, oldest sstables first, or
sstables you know overlap), you can likely manually clean things up pretty
quickly. At one point, I had a jar that would do single sstable at a time,
oldest sstable first, and it pretty much worked for this purpose most of
the time.

If you have room, a "nodetool compact" on stcs will also work, but it'll
give you one huge sstable, which will be unfortunate long term (probably
less of a problem if you're no longer writing to this table).


On Mon, Sep 10, 2018 at 10:29 AM Charulata Sharma (charshar)
<charshar@cisco.com.invalid> wrote:

> Scrub takes a very long time and does not remove the tombstones. You
> should do garbage cleaning. It immediately removes the tombstones.
>
>
>
> Thaks,
>
> Charu
>
>
>
> *From: *Oleksandr Shulgin <oleksandr.shulgin@zalando.de>
> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Date: *Monday, September 10, 2018 at 6:53 AM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *Drop TTLd rows: upgradesstables -a or scrub?
>
>
>
> Hello,
>
>
>
> We have some tables with significant amount of TTLd rows that have expired
> by now (and more gc_grace_seconds have passed since the TTL).  We have
> stopped writing more data to these tables quite a while ago, so background
> compaction isn't running.  The compaction strategy is the default
> SizeTiered one.
>
>
>
> Now we would like to get rid of all the droppable tombstones in these
> tables.  What would be the approach that puts the least stress on the
> cluster?
>
>
>
> We've considered a few, but the most promising ones seem to be these two:
> `nodetool scrub` or `nodetool upgradesstables -a`.  We are using Cassandra
> version 3.0.
>
>
>
> Now, this docs page recommends to use upgradesstables wherever possible:
> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsScrub.html
>
> What is the reason behind it?
>
>
>
> From source code I can see that Scrubber the class which is going to drop
> the tombstones (and report the total number in the logs):
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/Scrubber.java#L308
>
>
>
> I couldn't find similar handling in the upgradesstables code path.  Is the
> assumption correct that this one will not drop the tombstone as a side
> effect of rewriting the files?
>
>
>
> Any drawbacks of using scrub for this task?
>
>
>
> Thanks,
> --
>
> Oleksandr "Alex" Shulgin | Senior Software Engineer | Team Flux | Data
> Services | Zalando SE | Tel: +49 176 127-59-707
>
>
>

Mime
View raw message