From user-return-64072-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Thu Jun 20 06:13:57 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 05970180670 for ; Thu, 20 Jun 2019 08:13:56 +0200 (CEST) Received: (qmail 83821 invoked by uid 500); 20 Jun 2019 06:13:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 83809 invoked by uid 99); 20 Jun 2019 06:13:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jun 2019 06:13:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 0BA991A439F for ; Thu, 20 Jun 2019 06:13:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.801 X-Spam-Level: * X-Spam-Status: No, score=1.801 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=mailjet.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id nja7HZSCqknE for ; Thu, 20 Jun 2019 06:13:46 +0000 (UTC) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 22D4460CF6 for ; Thu, 20 Jun 2019 06:13:45 +0000 (UTC) Received: by mail-io1-f43.google.com with SMTP id k20so163848ios.10 for ; Wed, 19 Jun 2019 23:13:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=JKicbupJaO3uSAE97vfp4ZpzWjVTn9C753xzC6RZ8xA=; b=t1D6XcZeFETgBsp5u1Jrd0VkErjpzE9JVmGQnWAlkgz0oGgkRY/TISW8Ud2U7c6nl3 PeJutlzSioZvZbGP06DlIR1SS6er/8FSB6F1wJd9ovzPiTp/wfFyUeL5Bhta5rdz32WI XxJhEG5UgsWI2NMghK23kbuPoX6tebQ/KFVauQQnltpUbW22w/W4AYuRr+KzPhHUO6hJ PqXr3IlkTQKdKHsyJ7RDG7AHVgo7/WXFgV5np3W2PZpeB9ADtQNgC7Ilrpdr9Eikre/g MeHkCo2phS9tetq/mP2YAV7cIqHMb0Tdsr6uZiVzWfKMLv5zkCDznc4TNe3L/1mZso3s T4tA== X-Gm-Message-State: APjAAAUyuw+0QpQYoBwcAFuwGuMPsJDW2F0DDyUlAm2Wovca4gnn/bHv Rck3PpcalMxgl4vEQ4HkbEmjRroSsSkvczd7v2jzdjdn/4o= X-Google-Smtp-Source: APXvYqxOhyDyYr94UXemQoWIjYS1v9LWwhLEKbtCfFpMduzAxD9dT8HQ7rXZewkG8cAH3C4skm7WcAXu/UYcCijDpPo= X-Received: by 2002:a6b:f711:: with SMTP id k17mr11984700iog.273.1561011223609; Wed, 19 Jun 2019 23:13:43 -0700 (PDT) MIME-Version: 1.0 References: <37A66F25-E2C4-4BAF-B23F-A522D4E61ACB@gmail.com> In-Reply-To: From: =?UTF-8?Q?L=C3=A9o_FERLIN_SUTTON?= Date: Thu, 20 Jun 2019 08:13:32 +0200 Message-ID: Subject: Re: Tombstones not getting purged To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000795da9058bbb3fd4" --000000000000795da9058bbb3fd4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jun 20, 2019 at 7:37 AM Alexander Dejanovski wrote: > Hi Leo, > > The overlapping SSTables are indeed the most probable cause as suggested > by Jeff. > Do you know if the tombstone compactions actually triggered? (did the > SSTables name change?) > Hello ! I believe they have changed. I do not remember the sstable name but the "last modified" has changed recently for these tables. > Could you run the following command to list SSTables and provide us the > output? It will display both their timestamp ranges along with the > estimated droppable tombstones ratio. > > > for f in *Data.db; do meta=3D$(sstablemetadata -gc_grace_seconds 259200 $= f); > echo $(date --date=3D@$(echo "$meta" | grep Maximum\ time | cut -d" " -f= 3| > cut -c 1-10) '+%m/%d/%Y %H:%M:%S') $(date --date=3D@$(echo "$meta" | grep > Minimum\ time | cut -d" " -f3| cut -c 1-10) '+%m/%d/%Y %H:%M:%S') $(echo > "$meta" | grep droppable) $(ls -lh $f); done | sort > Here is the results : ``` 04/01/2019 22:53:12 03/06/2018 16:46:13 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 16G Apr 13 14:35 md-147916-big-Data.db 04/11/2262 23:47:16 10/09/1940 19:13:17 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 218M Jun 20 05:57 md-167948-big-Data.db 04/11/2262 23:47:16 10/09/1940 19:13:17 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 2.2G Jun 20 05:57 md-167942-big-Data.db 05/01/2019 08:03:24 03/06/2018 16:46:13 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 4.6G May 1 08:39 md-152253-big-Data.db 05/09/2018 06:35:03 03/06/2018 16:46:07 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 30G Apr 13 22:09 md-147948-big-Data.db 05/21/2019 05:28:01 03/06/2018 16:46:16 Estimated droppable tombstones: 0.45150604672159905 -rw-r--r-- 1 cassandra cassandra 1.1G Jun 20 05:55 md-167943-big-Data.db 05/22/2019 11:54:33 03/06/2018 16:46:16 Estimated droppable tombstones: 0.30826566640798975 -rw-r--r-- 1 cassandra cassandra 7.6G Jun 20 04:35 md-167913-big-Data.db 06/13/2019 00:02:40 03/06/2018 16:46:08 Estimated droppable tombstones: 0.20980847354256815 -rw-r--r-- 1 cassandra cassandra 6.9G Jun 20 04:51 md-167917-big-Data.db 06/17/2019 05:56:12 06/16/2019 20:33:52 Estimated droppable tombstones: 0.6114260192855792 -rw-r--r-- 1 cassandra cassandra 257M Jun 20 05:29 md-167938-big-Data.db 06/18/2019 11:21:55 03/06/2018 17:48:22 Estimated droppable tombstones: 0.18655813086540254 -rw-r--r-- 1 cassandra cassandra 2.2G Jun 20 05:52 md-167940-big-Data.db 06/19/2019 16:53:04 06/18/2019 11:22:04 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 425M Jun 19 17:08 md-167782-big-Data.db 06/20/2019 04:17:22 06/19/2019 16:53:04 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 146M Jun 20 04:18 md-167921-big-Data.db 06/20/2019 05:50:23 06/20/2019 04:17:32 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 42M Jun 20 05:56 md-167946-big-Data.db 06/20/2019 05:56:03 06/20/2019 05:50:32 Estimated droppable tombstones: 0.0 -rw-r--r-- 2 cassandra cassandra 4.8M Jun 20 05:56 md-167947-big-Data.db 07/03/2018 17:26:54 03/06/2018 16:46:07 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 27G Apr 13 17:45 md-147919-big-Data.db 09/09/2018 18:55:23 03/06/2018 16:46:08 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 30G Apr 13 18:57 md-147926-big-Data.db 11/30/2018 11:52:33 03/06/2018 16:46:08 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 14G Apr 13 13:53 md-147908-big-Data.db 12/20/2018 07:30:03 03/06/2018 16:46:08 Estimated droppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 9.3G Apr 13 13:28 md-147906-big-Data.db ``` You could also check the min and max tokens in each SSTable (not sure if > you get that info from 3.0 sstablemetadata) so that you can detect the > SSTables that overlap on token ranges with the ones that carry the > tombstones, and have earlier timestamps. This way you'll be able to trigg= er > manual compactions, targeting those specific SSTables. > I have checked and I don't believe the info is available in the 3.0.X version of sstablemetadata :( > The rule for a tombstone to be purged is that there is no SSTable outside > the compaction that would possibly contain the partition and that would > have older timestamps. > Is there a way to log these checks and decisions made by the compaction thread ? > Is this a followup on your previous issue where you were trying to perfor= m > a major compaction on an LCS table? > In some way. We are trying to globally reclaim the data used up by our tombstones (on more than one table). We have recently started to purge old data in our cassandra cluster, and since (on cloud providers) `Disk space isn't cheap` we are trying to be sure the data correctly expires and the disk space is reclaimed ! The major compaction on the LCS table was one of our unsuccessful attempts (too long and too much disk space used, so abandoned), and we are currently trying to tweak the compaction parameters to speed things up. Regards. Leo On Thu, Jun 20, 2019 at 7:02 AM Jeff Jirsa wrote: > >> Probably overlapping sstables >> >> Which compaction strategy? >> >> >> > On Jun 19, 2019, at 9:51 PM, L=C3=A9o FERLIN SUTTON >> wrote: >> > >> > I have used the following command to check if I had droppable >> tombstones : >> > `/usr/bin/sstablemetadata --gc_grace_seconds 259200 >> /var/lib/cassandra/data/stats/tablename/md-sstablename-big-Data.db` >> > >> > I checked every sstable in a loop and had 4 sstables with droppable >> tombstones : >> > >> > ``` >> > Estimated droppable tombstones: 0.1558453651124074 >> > Estimated droppable tombstones: 0.20980847354256815 >> > Estimated droppable tombstones: 0.30826566640798975 >> > Estimated droppable tombstones: 0.45150604672159905 >> > ``` >> > >> > I changed my compaction configuration this morning (via JMX) to force = a >> tombstone compaction. These are my settings on this node : >> > >> > ``` >> > { >> > "max_threshold":"32", >> > "min_threshold":"4", >> > "unchecked_tombstone_compaction":"true", >> > "tombstone_threshold":"0.1", >> > >> "class":"org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy= " >> > } >> > ``` >> > The threshold is lowed than the amount of tombstones in these sstables >> and I expected the setting `unchecked_tombstone_compaction=3DTrue` would >> force cassandra to run a "Tombstone Compaction", yet about 24h later all >> the tombstones are still there. >> > >> > ## About the cluster : >> > >> > The compaction backlog is clear and here are our cassandra settings : >> > >> > Cassandra 3.0.18 >> > concurrent_compactors: 4 >> > compaction_throughput_mb_per_sec: 150 >> > sstable_preemptive_open_interval_in_mb: 50 >> > memtable_flush_writers: 4 >> > >> > >> > Any idea what I might be missing ? >> > >> > Regards, >> > >> > Leo >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org >> For additional commands, e-mail: user-help@cassandra.apache.org >> >> --000000000000795da9058bbb3fd4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, Jun 20, 2019 at 7:37 AM Alexander= Dejanovski <alex@thelastpickl= e.com> wrote:
Hi Leo,
The overlapping SSTables are indeed the most probable cause as sugg= ested by Jeff.
Do you know if the tombstone compactions actually trigge= red? (did the SSTables name change?)

Hello !

I believe they have changed. I do no= t remember the sstable name but the "last modified" has changed r= ecently for these tables.=C2=A0
=C2=A0
Could you run= the following command to list SSTables and provide us the output? It will = display both their timestamp ranges along with the estimated droppable tomb= stones ratio.=C2=A0


for f in *Data.db;= do meta=3D$(sstablemetadata -gc_grace_seconds 259200 $f); echo $(date --da= te=3D@$(echo "$meta" | grep Maximum\ time | cut -d" "=C2=A0 = -f3| cut -c 1-10) '+%m/%d/%Y %H:%M:%S') $(date --date=3D@$(e= cho "$meta" | grep Minimum\ time | cut -d" "=C2=A0 -f= 3| cut -c 1-10) '+%m/%d/%Y %H:%M:%S') $(echo "$meta" | gr= ep droppable) $(ls -lh $f); done | sort

=

Here is the= results :

```
04/01/20= 19 22:53:12 03/06/2018 16:46:13 Estimated droppable tombstones: 0.0 -rw-r--= r-- 1 cassandra cassandra 16G Apr 13 14:35 md-147916-big-Data.db
04/11/2= 262 23:47:16 10/09/1940 19:13:17 Estimated droppable tombstones: 0.0 -rw-r-= -r-- 1 cassandra cassandra 218M Jun 20 05:57 md-167948-big-Data.db
04/11= /2262 23:47:16 10/09/1940 19:13:17 Estimated droppable tombstones: 0.0 -rw-= r--r-- 1 cassandra cassandra 2.2G Jun 20 05:57 md-167942-big-Data.db
05/= 01/2019 08:03:24 03/06/2018 16:46:13 Estimated droppable tombstones: 0.0 -r= w-r--r-- 1 cassandra cassandra 4.6G May 1 08:39 md-152253-big-Data.db
05= /09/2018 06:35:03 03/06/2018 16:46:07 Estimated droppable tombstones: 0.0 -= rw-r--r-- 1 cassandra cassandra 30G Apr 13 22:09 md-147948-big-Data.db
0= 5/21/2019 05:28:01 03/06/2018 16:46:16 Estimated droppable tombstones: 0.45= 150604672159905 -rw-r--r-- 1 cassandra cassandra 1.1G Jun 20 05:55 md-16794= 3-big-Data.db
05/22/2019 11:54:33 03/06/2018 16:46:16 Estimated droppabl= e tombstones: 0.30826566640798975 -rw-r--r-- 1 cassandra cassandra 7.6G Jun= 20 04:35 md-167913-big-Data.db
06/13/2019 00:02:40 03/06/2018 16:46:08 = Estimated droppable tombstones: 0.20980847354256815 -rw-r--r-- 1 cassandra = cassandra 6.9G Jun 20 04:51 md-167917-big-Data.db
06/17/2019 05:56:12 06= /16/2019 20:33:52 Estimated droppable tombstones: 0.6114260192855792 -rw-r-= -r-- 1 cassandra cassandra 257M Jun 20 05:29 md-167938-big-Data.db
06/18= /2019 11:21:55 03/06/2018 17:48:22 Estimated droppable tombstones: 0.186558= 13086540254 -rw-r--r-- 1 cassandra cassandra 2.2G Jun 20 05:52 md-167940-bi= g-Data.db
06/19/2019 16:53:04 06/18/2019 11:22:04 Estimated droppable to= mbstones: 0.0 -rw-r--r-- 1 cassandra cassandra 425M Jun 19 17:08 md-167782-= big-Data.db
06/20/2019 04:17:22 06/19/2019 16:53:04 Estimated droppable = tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 146M Jun 20 04:18 md-16792= 1-big-Data.db
06/20/2019 05:50:23 06/20/2019 04:17:32 Estimated droppabl= e tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 42M Jun 20 05:56 md-1679= 46-big-Data.db
06/20/2019 05:56:03 06/20/2019 05:50:32 Estimated droppab= le tombstones: 0.0 -rw-r--r-- 2 cassandra cassandra 4.8M Jun 20 05:56 md-16= 7947-big-Data.db
07/03/2018 17:26:54 03/06/2018 16:46:07 Estimated dropp= able tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 27G Apr 13 17:45 md-1= 47919-big-Data.db
09/09/2018 18:55:23 03/06/2018 16:46:08 Estimated drop= pable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 30G Apr 13 18:57 md-= 147926-big-Data.db
11/30/2018 11:52:33 03/06/2018 16:46:08 Estimated dro= ppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 14G Apr 13 13:53 md= -147908-big-Data.db
12/20/2018 07:30:03 03/06/2018 16:46:08 Estimated dr= oppable tombstones: 0.0 -rw-r--r-- 1 cassandra cassandra 9.3G Apr 13 13:28 = md-147906-big-Data.db
```=C2=A0

You could also = check the min and max tokens in each SSTable (not sure if you get that info= from 3.0 sstablemetadata) so that you can detect the SSTables that overlap= on token ranges with the ones that carry the tombstones, and have earlier = timestamps. This way you'll be able to trigger manual compactions, targ= eting those specific SSTables.


I have checked and I don't believe the info is available in th= e 3.0.X version of sstablemetadata :(
=C2=A0
The rule for a tom= bstone to be purged is that there is no SSTable outside the compaction that= would possibly contain the partition and that would have older timestamps.=
=C2=A0Is there a way to log these checks and = decisions made by the compaction thread ?
=C2=A0
Is = this a followup on your previous issue where you were trying to perform a m= ajor compaction on an LCS table?

In some way.=C2=A0

We are trying to globally= reclaim the data used up by our tombstones (on more than one table). We ha= ve recently started to purge old data in our cassandra cluster, and since (= on cloud providers) `Disk space isn't cheap` we are trying to be sure t= he data correctly expires and the disk space is reclaimed !

<= /div>
The major compaction on the LCS table was one of our unsuccessful= attempts (too long and too much disk space used, so abandoned), and we are= currently trying to tweak the compaction parameters to speed things up.

Regards.

Leo

=
On Thu, Jun 20, 2019 at 7:02 = AM Jeff Jirsa <jji= rsa@gmail.com> wrote:
Probably overlapping sstables

Which compaction strategy?


> On Jun 19, 2019, at 9:51 PM, L=C3=A9o FERLIN SUTTON <lferlin@mailje= t.com.invalid> wrote:
>
> I have used the following command to check if I had droppable tombston= es :
> `/usr/bin/sstablemetadata --gc_grace_seconds 259200 /var/lib/cassandra= /data/stats/tablename/md-sstablename-big-Data.db`
>
> I checked every sstable in a loop and had 4 sstables with droppable to= mbstones :
>
> ```
> Estimated droppable tombstones: 0.1558453651124074
> Estimated droppable tombstones: 0.20980847354256815
> Estimated droppable tombstones: 0.30826566640798975
> Estimated droppable tombstones: 0.45150604672159905
> ```
>
> I changed my compaction configuration this morning (via JMX) to force = a tombstone compaction. These are my settings on this node :
>
> ```
> {
> "max_threshold":"32",
> "min_threshold":"4",
> "unchecked_tombstone_compaction":"true",
> "tombstone_threshold":"0.1",
> "class":"org.apache.cassandra.db.compaction.SizeTieredC= ompactionStrategy"
> }
> ```
> The threshold is lowed than the amount of tombstones in these sstables= and I expected the setting `unchecked_tombstone_compaction=3DTrue` would f= orce cassandra to run a "Tombstone Compaction", yet about 24h lat= er all the tombstones are still there.
>
> ## About the cluster :
>
> The compaction backlog is clear and here are our cassandra settings : =
>
> Cassandra 3.0.18
> concurrent_compactors: 4
> compaction_throughput_mb_per_sec: 150
> sstable_preemptive_open_interval_in_mb: 50
> memtable_flush_writers: 4
>
>
> Any idea what I might be missing ?
>
> Regards,
>
> Leo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org

--000000000000795da9058bbb3fd4--