Hi there,

I am running a 6 node cluster on 2.1.7 with a table using DTCS to store time series data, for up to 12 hours (using ttl of 12h). Data are written as they arrive, without any update or active delete.
During the first 12h, the cluster gets filled with data. And a bit later, the amount of data stored remain stable. I naturally observe that the oldest SSTables are not much older than 12h. However, at some point, a few nodes started to get there data larger growint at the same rate as the initial filling. Looking at the data dir,I observed that on those nodes, SSTables that should have been dropped are still around (unlike on the other nodes).
I dropped the keyspace, and started from scratch. Again, a node started to store more data. I waited longer, and at some points, obsolete sstables got dropped.
Now I don't understand why there is this behaviour, and I was hopping someone could shade some light on this. Or maybe it is a bug? Anyone?


For info, the compactaion parameters I am using is:
compaction = {'min_threshold': '16', 'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy', 'max_sstable_age_days': '0.05', 'base_time_seconds': '300', 'max_threshold': '16'}
with a gc-grace-period of 0.


And attached is a plot on opscenter.

Thanks!

Christophe