cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-11060) Allow DTCS old SSTable filtering to use min timestamp instead of max
Date Mon, 25 Jan 2016 09:52:40 GMT


Marcus Eriksson commented on CASSANDRA-11060:

also see CASSANDRA-11056

> Allow DTCS old SSTable filtering to use min timestamp instead of max
> --------------------------------------------------------------------
>                 Key: CASSANDRA-11060
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Sam Bisbee
>              Labels: dtcs
> We have observed a DTCS behavior when using TTLs where SSTables are never or very rarely
fully expired due to compaction, allowing expired data to be "stuck" in large partially expired
> This is because compaction filtering is performed on the max timestamp, which continues
to grow as SSTables are compacted together. This means they will never move past max_sstable_age_days.
With a sufficiently large TTL, like 30 days, this allows old but not expired SSTables to continue
combining and never become fully expired, even with a max_sstable_age_days of 1.
> As a result we have seen expired data hang around in large SSTables for over six months
longer than it should have. This is obviously wasteful and a disk capacity issue.
> As a result we have been running an extended version of DTCS called MTCS in some deployments.
The only change is that it uses min timestamp instead of max for compaction filtering (filterOldSSTables()).
This allows SSTables to move beyond max_sstable_age_days and stop compacting, which means
the entire SSTable can become fully expired and be dropped off disk as intended.
> You can see and test MTCS here:
> I am not advocating that MTCS be its own stand alone compaction strategy. However, I
would like to see a configuration option for DTCS that allows you to specify whether old SSTables
should be filtered on min or max timestamp.

This message was sent by Atlassian JIRA

View raw message