Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1D7E1200CEF for ; Mon, 4 Sep 2017 13:57:20 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1BF15164C3A; Mon, 4 Sep 2017 11:57:20 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E1986164C34 for ; Mon, 4 Sep 2017 13:57:18 +0200 (CEST) Received: (qmail 52267 invoked by uid 500); 4 Sep 2017 11:57:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 52257 invoked by uid 99); 4 Sep 2017 11:57:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Sep 2017 11:57:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BA7D7C1F4E for ; Mon, 4 Sep 2017 11:57:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.391 X-Spam-Level: X-Spam-Status: No, score=-0.391 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id ZTvyrzWdYINb for ; Mon, 4 Sep 2017 11:57:11 +0000 (UTC) Received: from mail-oi0-f47.google.com (mail-oi0-f47.google.com [209.85.218.47]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B39186129E for ; Mon, 4 Sep 2017 11:57:02 +0000 (UTC) Received: by mail-oi0-f47.google.com with SMTP id t75so2087004oie.3 for ; Mon, 04 Sep 2017 04:57:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Rw5x7+uLbgNEBn85QA6Tis0ABxijxDL1O9gVtxsF2l0=; b=hrISLaOzMlGLUPgcJu4pi9jeEsH8ZTYIuJpRAChq20wdboov09KW9vcUcdZ2D3UOZe 43Z2apecBTHgr9f4qTvMDhadzIo89/2GTfOcagJZ0S/cKO8xfjC4jPkGUpY9yZ6S56fu CSTQPulHFGpdoBZyMyaqyeiFL63h1SwVHsD94k9XTgmSvVmHdlQWcdCVqtFBFHK+kRoY sTmIB4sTqhpqhNYLabp1PtrZnbnyHDDvxII51sIsDgCNIfM18yon2NnI6kc+H65Y8oiM 8BMvBm7AiSHRhPpINrdzorHHoJCQ9+IRMFrcRN3zXbnDenYJJGorQVoWmIlQmumT59WL C3FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Rw5x7+uLbgNEBn85QA6Tis0ABxijxDL1O9gVtxsF2l0=; b=AzD+p23WCYMkrj1x/f/sCd/dYJpI2JVCA601PoAx1x0ppr9DCyA5QKxZzBlNASXv4A wvgdtA4n8PahJRAsS9zal/rZ09fcvD4KOUPTvqdfM3XCbTIKqKLG3n63MlKigg7uWkp5 q62R40gCtgAwwPZ+buuXzpL0xMs47YR7zqmImGttGwgDb9coRH5fWEF12wuWxMJwpOoa JWOlDcRKKm9PGpPknDhSQUs2/YhnUOVT7cuv/9Ne56KgaQ++N6uKhboQoq5ifJXditw8 eRseCxKl1A7L59Uy7ZC8/Dym1mGmKOho+oG/IYiyPARttsoD71xRMw11pqcY1Mh79gxg nYvQ== X-Gm-Message-State: AHPjjUipoIJ3gLDj2BAjMi8x3Jk1CROXDuxvpoMwSyWn3pUecSDz7cj3 Ynu/XpTRxhs0NqhTXfoXKJSzQOcY3yE4Nc0= X-Google-Smtp-Source: ADKCNb6DitNUzfNI1bGQHTGACQkamW7euZNoNEBSukXyGt0oIU00WEkm/s6KBKkD8W4388thluDhRhxykS4HzQ42xkk= X-Received: by 10.202.83.209 with SMTP id h200mr430562oib.8.1504526219929; Mon, 04 Sep 2017 04:56:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.88.8 with HTTP; Mon, 4 Sep 2017 04:56:39 -0700 (PDT) In-Reply-To: References: <2DF54BB2-2A0C-4F56-8342-8BEF3AFEB935@gmail.com> <7210840B-6C97-4DA0-9FEA-79AF3D18F165@gmail.com> <34A86717-039F-4760-8521-8F11C11E104C@gmail.com> <6622EEAF-096C-406D-AAA9-1D10881C9484@gmail.com> <5CCFE4AD-9D45-4EF7-891C-8BFB4112D493@gmail.com> <7D533B73-0EE2-455D-9B89-08A73E049C6B@gmail.com> From: Nicolas Guyomar Date: Mon, 4 Sep 2017 13:56:39 +0200 Message-ID: Subject: Re: old big tombstone data file occupy much disk space To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="001a113d5ea2e4ee1405585bcee8" archived-at: Mon, 04 Sep 2017 11:57:20 -0000 --001a113d5ea2e4ee1405585bcee8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Wrong copy/paste ! Looking at the code, it should do nothing : // look up the sstables now that we're on the compaction executor, so we don't try to re-compact // something that was already being compacted earlier. On 4 September 2017 at 13:54, Nicolas Guyomar wrote: > You'll get the WARN "Will not compact {}: it is not an active sstable" := ) > > On 4 September 2017 at 12:07, Shalom Sagges > wrote: > >> By the way, does anyone know what happens if I run a user defined >> compaction on an sstable that's already in compaction? >> >> >> >> >> >> >> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges >> wrote: >> >>> Try this blog by The Last Pickle: >>> >>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html >>> >>> >>> >>> >>> >>> >>> Shalom Sagges >>> DBA >>> >>> We Create Meaningful Connection= s >>> >>> >>> >>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa wrote: >>> >>>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWC= S >>>> was designed for ttl-only time series use cases >>>> >>>> Alternatively, if you have IO to spare, you may find LCS works as well >>>> (it'll cause quite a bit more compaction, but a much higher chance to >>>> compact away tombstones) >>>> >>>> There are also tombstone focused sub properties to more aggressively >>>> compact sstables that have a lot of tombstones - check the docs for >>>> "unchecked tombstone compaction" and "tombstone threshold" - enabling = those >>>> will enable more aggressive automatic single-sstable compactions >>>> >>>> -- >>>> Jeff Jirsa >>>> >>>> >>>> On Sep 2, 2017, at 7:10 AM, qf zhou wrote: >>>> >>>> >>>> Yes, your are right. I am using STCS compaction strategy with some kin= d >>>> of timeseries model. Too much disk space has been occupied. >>>> >>>> What should I do to stop the disk full ? >>>> >>>> I only want to keep 100 days data most recently, so I set default_ti= me_to_live >>>> =3D 8640000(100 days ). >>>> >>>> I know I need to do something to stop the disk space cost, but I reall= y >>>> don=E2=80=99t know how to do it. >>>> >>>> >>>> Here is the strategy of the big data table : >>>> >>>> AND compaction =3D {'class': 'org.apache.cassandra.db.compa >>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32', >>>> 'min_threshold': '12', 'tombstone_threshold': '0.1', >>>> 'unchecked_tombstone_compaction': 'true'} >>>> AND compression =3D {'chunk_length_in_kb': '64', 'class': ' >>>> org.apache.cassandra.io.compress.LZ4Compressor'} >>>> AND crc_check_chance =3D 1.0 >>>> AND dclocal_read_repair_chance =3D 0.1 >>>> AND default_time_to_live =3D 8640000 >>>> AND gc_grace_seconds =3D 432000 >>>> >>>> >>>> >>>> =E5=9C=A8 2017=E5=B9=B49=E6=9C=882=E6=97=A5=EF=BC=8C=E4=B8=8B=E5=8D=88= 7:34=EF=BC=8CNicolas Guyomar =E5=86=99=E9=81=93= =EF=BC=9A >>>> >>>> your are using STCS compaction strategy with some kind of timeseries >>>> model, and you are going to end up with yor disk full! >>>> >>>> >>>> >>> >> >> This message may contain confidential and/or privileged information. >> If you are not the addressee or authorized to receive this on behalf of >> the addressee you must not use, copy, disclose or take action based on t= his >> message or any information herein. >> If you have received this message in error, please advise the sender >> immediately by reply email and delete this message. Thank you. >> > > --001a113d5ea2e4ee1405585bcee8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Wrong copy/paste !=C2=A0

Looking at the= code, it should do nothing :=C2=A0

=C2=A0// = look up the sstables now that we're on the compaction executor, so we d= on't try to re-compact
=C2=A0// something that was already be= ing compacted earlier.

On 4 September 2017 at 13:54, Nicolas Guyomar <nicolas.guyomar@gmail.com> wrote:
You'll get the WARN=C2=A0"Will not com= pact {}: it is not an active sstable" =C2=A0:)=C2=A0

On 4 September 2017 at 12:07, Shalom Sagges = <shaloms@liv= eperson.com> wrote:
By the way, does anyone know what happens if I run a user defined= compaction on an sstable that's already in compaction?=C2=A0




=C2=A0

On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagge= s <shaloms@liveperson.com> wrote:
Try this blog by The Last Pickle:





=C2=A0
Shalom Sagges
DBA
We Cr= eate Meaningful Connections

=C2=A0

On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa <= span dir=3D"ltr"><= jjirsa@gmail.com> wrote:
If you're on 3.0 (3.0.6 or 3.0.8 or newer I don= 9;t remember which), TWCS was designed for ttl-only time series use cases

Alternatively, if you have IO to spare, you may fin= d LCS works as well (it'll cause quite a bit more compaction, but a muc= h higher chance to compact away tombstones)

There = are also tombstone focused sub properties to more aggressively compact ssta= bles that have a lot of tombstones - check the docs for "unchecked tom= bstone compaction" and "tombstone threshold" - enabling thos= e will enable more aggressive automatic single-sstable compactions=C2=A0

--=C2=A0
Jeff Jirsa


On Sep 2, 2= 017, at 7:10 AM, qf zhou <zhouqf2013@gmail.com> wrote:


Yes, your are right. I am using STCS compaction strategy with some= kind of timeseries model. Too much disk space has been occupied.

What=C2=A0should I =C2=A0d= o to stop =C2=A0the =C2=A0disk full ?=C2=A0

=C2=A0I only want to keep 100 days data most rec= ently, =C2=A0so I set=C2=A0default_time_to_live =3D 8640000(100 days= ).

I know I need to do something to stop the disk= space cost, but I really don=E2=80=99t know how to do it.


Here is the strategy of the big data table :<= /span>

=
=C2=A0 =C2=A0 AND compaction =3D {'class': 'org= .apache.cassandra.db.compaction.SizeTieredCompactionStrategy'= , 'max_threshold': '32', 'min_threshold': '12&#= 39;, 'tombstone_threshold': '0.1', 'unchecked_tombstone= _compaction': 'true'}
=C2=A0 =C2=A0 AND compress= ion =3D {'chunk_length_in_kb': '64', 'class': '= org.apache.cas= sandra.io.compress.LZ4Compressor'}
=C2=A0 =C2=A0 AND= crc_check_chance =3D 1.0
=C2=A0 =C2=A0 AND dclocal_read_repair_c= hance =3D 0.1
=C2=A0 =C2=A0 AND default_time_to_live =3D 8640000<= /div>
=C2=A0 =C2=A0 AND gc_grace_seconds =3D 432000



=E5=9C=A8 2017=E5=B9=B49=E6=9C=882=E6=97=A5=EF=BC=8C=E4= =B8=8B=E5=8D=887:34=EF=BC=8CNicolas Guyomar <nicolas.guyomar@gmail.com> =E5= =86=99=E9=81=93=EF=BC=9A

your are u= sing STCS compaction strategy with some kind of timeseries model, and you a= re going to end up with yor disk full!

<= /div>



This message may contain confidential and/or privileg= ed information.=C2=A0
If you are not the = addressee or authorized to receive this on behalf of the addressee you must= not use, copy, disclose or take action based on this message or any inform= ation herein.=C2=A0
If you have received = this message in error, please advise the sender immediately by reply email = and delete this message. Thank you.


--001a113d5ea2e4ee1405585bcee8--