cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Kinder <dkin...@turnitin.com>
Subject Re: Overwhelming tombstones with LCS
Date Fri, 10 Jul 2015 17:57:47 GMT
On Sun, Jul 5, 2015 at 1:40 PM, Roman Tkachenko <roman@mailgunhq.com> wrote:

> Hey guys,
>
> I have a table with RF=3 and LCS. Data model makes use of "wide rows". A
> certain query run against this table times out and tracing reveals the
> following error on two out of three nodes:
>
> *Scanned over 100000 tombstones; query aborted (see
> tombstone_failure_threshold)*
>
> This basically means every request with CL higher than "one" fails.
>
> I have two questions:
>
> * How could it happen that only two out of three nodes have overwhelming
> tombstones? For the third node tracing shows sensible *"Read 815 live and
> 837 tombstoned cells"* traces.
>

One theory: before 2.1.6 compactions on wide rows with lots of tombstones
could take forever or potentially never finish. What version of Cassandra
are you on? It may be that you got lucky with one node that has been able
to keep up but the others haven't been able to.


>
> * Anything I can do to fix those two nodes? I have already set gc_grace to
> 1 day and tried to make compaction strategy more aggressive
> (unchecked_tombstone_compaction - true, tombstone_threshold - 0.01) to no
> avail - a couple of days have already passed and it still gives the same
> error.
>

You probably want major compaction which is coming soon for LCS (
https://issues.apache.org/jira/browse/CASSANDRA-7272) but not here yet.

The alternative is, if you have enough time and headroom (this is going to
do some pretty serious compaction so be careful), alter your table to STCS,
let it compact into one SSTable, then convert back to LCS. It's pretty
heavy-handed but as long as your gc_grace is low enough it'll do the job.
Definitely do NOT do this if you have many tombstones in single wide rows
and are not >2.1.6


>
> Thanks!
>
> Roman
>
>


-- 
Dan Kinder
Senior Software Engineer
Turnitin – www.turnitin.com
dkinder@turnitin.com

Mime
View raw message