Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 21 Aug 2017 10:56:00 +0000 (UTC)
From: "mck (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13061867.1491427038000.78193.1503312960657@Atlassian.JIRA>
In-Reply-To: <JIRA.13061867.1491427038000@Atlassian.JIRA>
References: <JIRA.13061867.1491427038000@Atlassian.JIRA> <JIRA.13061867.1491427038561@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore
 overlaps when dropping fully expired sstables
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 21 Aug 2017 10:56:15 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=
=3D16135028#comment-16135028 ]=20

mck edited comment on CASSANDRA-13418 at 8/21/17 10:55 AM:
-----------------------------------------------------------

[~rgerard],

Initial review comments:=20
 - {{CHANGES.txt}} needs a line
 - is this suitable for 3.11.x as well? (Does it constitute a stability pat=
ch? i think so, but that's an opinion)
 - is the method {{TWCSCompactionController.getFullyExpiredSSTables(..)}} r=
eally needed? (CompactionController:170 seems to do what we need already, o=
r am i missing something?)
 - {{CompactionController.ignoreOverlaps()}} is a bit of a concept, it dese=
rves some apidoc love.

Trivial stuff=E2=80=A6
 - CompactionController:108 missing space in {{if(}}, same on :170
 - CompactionController:232 any reason not to return an immutable set?
 - TWCSCompactionController:29 needs a blank line after
 - TWCSCompactionController:33 java declaration order. (static fields go be=
fore member fields)
 - TimeWindowCompactionStrategy:70  missing space in {{if(}}


was (Author: michaelsembwever):
[~rgerard],

Initial review comments:=20
 - {{CHANGES.txt}} needs a line
 - is this suitable for 3.11.x as well? (Does it constitute a stability pat=
ch? i think so, but that's an opinion)
 - is the method {{TWCSCompactionController.getFullyExpiredSSTables(..)}} r=
eally needed? (CompactionController:170 seems to do what we need already)
 - {{CompactionController.ignoreOverlaps()}} is a bit of a concept, it dese=
rves some apidoc love.

Trivial stuff=E2=80=A6
 - CompactionController:108 missing space in {{if(}}, same on :170
 - CompactionController:232 any reason not to return an immutable set?
 - TWCSCompactionController:29 needs a blank line after
 - TWCSCompactionController:33 java declaration order. (static fields go be=
fore member fields)
 - TimeWindowCompactionStrategy:70  missing space in {{if(}}

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-13418
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1341=
8
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Compaction
>            Reporter: Corentin Chary
>              Labels: twcs
>         Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well=
. If you really want read-repairs you're going to have sstables blocking th=
e expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction =3D true or tombstone_threshol=
d to a very low value and that will purge the blockers of old data that sho=
uld already have expired, thus removing the overlaps and allowing the other=
 SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you ha=
ve time series, you might not care if all your data doesn't exactly expire =
at the right time, or if data re-appears for some time, as long as it gets =
deleted as soon as it can. And in this situation I believe it would be real=
ly beneficial to allow users to simply ignore overlapping SSTables when loo=
king for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my datas=
et, so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would=
 be enough to greatly reduce entropy of the most used data (and if you have=
 timeseries, you're likely to have a dashboard doing the same important que=
ries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try i=
t on our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues alr=
eady.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org