cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Romain GERARD (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
Date Tue, 29 Aug 2017 08:10:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144901#comment-16144901
] 

Romain GERARD edited comment on CASSANDRA-13418 at 8/29/17 8:09 AM:
--------------------------------------------------------------------

bq. 2. only enabling unsafe_aggressive_sstable_expiration

When looking for sstables expired, you will *ignore *the overlaps and only look locally if
the current sstable is eligible.
When looking for sstables to compact, you will *not ignore* the overlaps and look globally
if the current sstable is eligible.

bq. 1. enabling both
When looking for sstables expired, you will *ignore *the overlaps and only look locally if
the current sstable is eligible.
When looking for sstables to compact, you will *ignore* the overlaps and look globally if
the current sstable is eligible.


-----


I made a new version of the patch with uncheckedTombstoneCompaction disabled and a warning
message.
https://github.com/criteo-forks/cassandra/commit/800ab325cbf7d9d4d5e60e2b959918426e121815


the diff 

{noformat}
diff --git a/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
index 43c90c7042..d21222c484 100644
--- a/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
@@ -67,9 +67,9 @@ public class TimeWindowCompactionStrategy extends AbstractCompactionStrategy
         else
             logger.debug("Enabling tombstone compactions for TWCS");

-        if (this.options.ignoreOverlaps)
-            this.uncheckedTombstoneCompaction = true;
-
+        if(this.options.ignoreOverlaps && !this.uncheckedTombstoneCompaction) {
+            logger.warn("You are running with sstables overlapping checks disabled but without
unchecked tombstone compaction, check that this what you want");
+        }
     }
{noformat}



was (Author: rgerard):
bq. 2. only enabling unsafe_aggressive_sstable_expiration

When looking for sstables expired, you will *ignore *the overlaps and only look locally if
the current sstable is eligible.
When looking for sstables to compact, you will *not ignore* the overlaps and look globally
if the current sstable is eligible.

bq. 1. enabling both
When looking for sstables expired, you will *ignore *the overlaps and only look locally if
the current sstable is eligible.
When looking for sstables to compact, you will *ignore* the overlaps and look globally if
the current sstable is eligible.


I made a new version of the patch with uncheckedTombstoneCompaction disabled and a warning
message.
https://github.com/criteo-forks/cassandra/commit/800ab325cbf7d9d4d5e60e2b959918426e121815


the diff 

{noformat}
diff --git a/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
index 43c90c7042..d21222c484 100644
--- a/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
@@ -67,9 +67,9 @@ public class TimeWindowCompactionStrategy extends AbstractCompactionStrategy
         else
             logger.debug("Enabling tombstone compactions for TWCS");

-        if (this.options.ignoreOverlaps)
-            this.uncheckedTombstoneCompaction = true;
-
+        if(this.options.ignoreOverlaps && !this.uncheckedTombstoneCompaction) {
+            logger.warn("You are running with sstables overlapping checks disabled but without
unchecked tombstone compaction, check that this what you want");
+        }
     }
{noformat}


> Allow TWCS to ignore overlaps when dropping fully expired sstables
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-13418
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Compaction
>            Reporter: Corentin Chary
>              Labels: twcs
>         Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If you really
want read-repairs you're going to have sstables blocking the expiration of other fully expired
SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a very low
value and that will purge the blockers of old data that should already have expired, thus
removing the overlaps and allowing the other SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have time series,
you might not care if all your data doesn't exactly expire at the right time, or if data re-appears
for some time, as long as it gets deleted as soon as it can. And in this situation I believe
it would be really beneficial to allow users to simply ignore overlapping SSTables when looking
for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, so this
isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be enough to
greatly reduce entropy of the most used data (and if you have timeseries, you're likely to
have a dashboard doing the same important queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on our system
and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message