cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Bisbee (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-11060) Allow DTCS old SSTable filtering to use min timestamp instead of max
Date Fri, 22 Jan 2016 21:44:40 GMT
Sam Bisbee created CASSANDRA-11060:

             Summary: Allow DTCS old SSTable filtering to use min timestamp instead of max
                 Key: CASSANDRA-11060
             Project: Cassandra
          Issue Type: Improvement
            Reporter: Sam Bisbee

We have observed a DTCS behavior when using TTLs where SSTables are never or very rarely fully
expired due to compaction, allowing expired data to be "stuck" in large partially expired

This is because compaction filtering is performed on the max timestamp, which continues to
grow as SSTables are compacted together. This means they will never move past max_sstable_age_days.
With a sufficiently large TTL, like 30 days, this allows old but not expired SSTables to continue
combining and never become fully expired, even with a max_sstable_age_days of 1.

As a result we have seen expired data hang around in large SSTables for over six months longer
than it should have. This is obviously wasteful and a disk capacity issue.

As a result we have been running an extended version of DTCS called MTCS in some deployments.
The only change is that it uses min timestamp instead of max for compaction filtering (filterOldSSTables()).
This allows SSTables to move beyond max_sstable_age_days and stop compacting, which means
the entire SSTable can become fully expired and be dropped off disk as intended.

You can see and test MTCS here:

I am not advocating that MTCS be its own stand alone compaction strategy. However, I would
like to see a configuration option for DTCS that allows you to specify whether old SSTables
should be filtered on min or max timestamp.

This message was sent by Atlassian JIRA

View raw message