Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id F0790200B50 for ; Sat, 30 Jul 2016 02:04:17 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id EF230160AA6; Sat, 30 Jul 2016 00:04:17 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2701B160A79 for ; Sat, 30 Jul 2016 02:04:16 +0200 (CEST) Received: (qmail 72038 invoked by uid 500); 30 Jul 2016 00:04:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72028 invoked by uid 99); 30 Jul 2016 00:04:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Jul 2016 00:04:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1371AC914E for ; Sat, 30 Jul 2016 00:04:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 3PG-Ms7NtiUV for ; Sat, 30 Jul 2016 00:04:10 +0000 (UTC) Received: from mail-oi0-f41.google.com (mail-oi0-f41.google.com [209.85.218.41]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 452ED5F256 for ; Sat, 30 Jul 2016 00:04:09 +0000 (UTC) Received: by mail-oi0-f41.google.com with SMTP id l72so126973274oig.2 for ; Fri, 29 Jul 2016 17:04:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=A6xyvrnSRcv95BAyFKmI+mE9H2vBaLzhBcj8HhPZYsM=; b=c0fAgO1wH7W29OQTUm0qWT73GVaCXSzzszNAChY+PcRBO7K5gokk0Vt6r3iG92Jckf IN/sIHUqIu9VCOYldtrtEX1BrMGGvqA9YMf1BUo4/7eq2dl7rXTVEZBdIDhYdX5l+i4R Nx/pOP+8J6x+geTM5PV+5YBUBlJlySh/WEoMav3yThq1vQDCdDAlVGZDzSDLvYs4ZPB3 N3w9hn+VRSu2prlXor3IO6Q7WmKst6bu32bW++ZoyYCurp+VX9fLuTzVhgrpYxU6q4BD NzxctQMkWeqom1RZf4GO4zcx9UxYNyTJwIH4exlxgBEse4BRi62QbQuK/MuB3f6/UN5H iQ0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=A6xyvrnSRcv95BAyFKmI+mE9H2vBaLzhBcj8HhPZYsM=; b=L+z6SHJH6aQZ33XTT0TACo0eDyZQ7FZ1jVdaD3EET+GwX1TMl4VinhKzPByE8bAEXh ItFu0M14hgF3pkG6wAKMBkxfdAaIc9i7TFbrwVJtN3/eG059z4/LLSxTGSjg+gNQnj0k +FzTcqe9FlaIW+cbL7SfV8B9KYkejeE1Mb7LPpmGtWz6E832PFUI3mvs8vAZtV0KTkCM JBowrvIebVuX6SRIpwyKDSEjLNl+qj5FOCrVb9nsCr0175gFyLfn6GklbIJqxw75squx tE1fX+pEUvUIAaxsGZzuD2wbz29IQ31fwVEAPdip8tz9rmRp6eDkc28hOp1SO/KlDKVw LLMw== X-Gm-Message-State: AEkooutfndCEJsgW8H8j37XOfSowqP3gQWewGQmPpG+w9SWkYYu3w6p5VzFpuwSvchShwaSgl5C8GsgSq8NsSQ== X-Received: by 10.202.219.213 with SMTP id s204mr26370889oig.151.1469837047776; Fri, 29 Jul 2016 17:04:07 -0700 (PDT) MIME-Version: 1.0 References: <5572FE7D-45B2-4148-A359-A0613B340E0B@crowdstrike.com> In-Reply-To: From: Eric Stevens Date: Sat, 30 Jul 2016 00:03:58 +0000 Message-ID: Subject: Re: Re : Purging tombstones from a particular row in SSTable To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113d35d61c4b900538cf1b76 archived-at: Sat, 30 Jul 2016 00:04:18 -0000 --001a113d35d61c4b900538cf1b76 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I haven't tested that specifically, but I haven't bumped into any particular optimization that allows it to skip reading an sstable where the entire relevant partition has been row-tombstoned. It's possible that something like that could happen by examining min/max timestamps on sstables, and not reading from any sstable with a partition-level tombstone where the max timestamp is less than the timestamp of the partition tombstone. However that presumes that it can have read the tombstones from each sstable before it read the occluded data, which I don't think is likely. Such an optimization could be there, but I haven't noticed it if it is, though I'm certainly not an expert (more of a well informed novice). If someone wants to set me straight on this point I'd love to know about it. On Fri, Jul 29, 2016 at 2:37 PM DuyHai Doan wrote: > @Eric > > Very interesting example. But then what is the case of row (should I say > partition ?) tombstones ? > > Suppose that in your example, I issued a DELETE FROM foo WHERE pk=3D'a' > > With the same SELECT statement than before, would C* be clever enough to > skip reading at all the whole partition (let's limit the example to a > single SSTable) ? > > On Fri, Jul 29, 2016 at 7:00 PM, Eric Stevens wrote: > >> > Sai was describing a timeout, not a failure due to the 100 K tombstone >> limit from cassandra.yaml. But I still might be missing things about >> tombstones. >> >> The trouble with tombstones is not the tombstones themselves, rather it'= s >> that Cassandra has a lot of deleted data to read through in sstables in >> order to satisfy a query. Although if you range constrain your cluster = key >> in your query, the read path can optimize that read to start somewhere n= ear >> the correct head of your selected data, that is _not_ true for tombstone= d >> data. >> >> Consider this exercise: >> CREATE TABLE foo ( >> pk text, >> ck int, >> PRIMARY KEY ((pk), ck) >> ) >> INSERT INTO foo (pk,ck) VALUES ('a', 1) >> ... >> INSERT INTO foo (pk,ck) VALUES ('a', 100000) >> >> $ nodetool flush >> >> DELETE FROM foo WHERE pk=3D'a' AND ck < 99999 >> >> We've now written a single "tiny" (bytes-wise) range tombstone. >> >> Now try to select from that table: >> SELECT * FROM foo WHERE ck > 50000 LIMIT 1 >> pk | ck >> -- | ------ >> a | 100000 >> >> This has to read from the first sstable, skipping over 49999 records >> before it can locate the first non-tombstoned cell. >> >> The problem isn't the size of the tombstone, tombstones themselves are >> cheaper (again, bytes-wise) than standard columns because they don't >> involve any value for the cell. The problem is that the read path canno= t >> anticipate in advance what cells are going to be occluded by the tombsto= ne, >> and in order to satisfy the query it needs to read then discard a large >> number of deleted cells. >> >> The reason the thresholds exist in cassandra.yaml is to help guide users >> away from performance anti-patterns that come from selects which include= a >> large number of tombstoned cells. >> >> On Thu, Jul 28, 2016 at 11:08 PM Alain RODRIGUEZ >> wrote: >> >>> Hi, >>> >>> @Eric >>> >>> Large range tombstones can occupy just a few bytes but can occlude >>>> millions of records, and have the corresponding performance impact on >>>> reads. It's really not the size of the tombstone on disk that matters= , but >>>> the number of records it occludes. >>> >>> >>> Sai was describing a timeout, not a failure due to the 100 K tombstone >>> limit from cassandra.yaml. But I still might be missing things about >>> tombstones. >>> >>> The read queries are continuously failing though because of the >>>> tombstones. "Request did not complete within rpc_timeout." >>>> >>> >>> So that is what looks weird to me. Reading 220 kb, even holding >>> tombstone should probably not take that long... Or am I wrong or missin= g >>> something? >>> >>> Your talk looks like cool stuff :-). >>> >>> @Sai >>> >>> The issues here was that tombstones were not in the SSTable, but rather >>>> in the Memtable >>> >>> >>> This sounds weird to me as well, knowing that memory is faster than dis= k >>> and that memtables are mutable data (so less stuff to read from there). >>> Flushing might have triggered some compaction, removing tombstones thou= gh. >>> >>> This still sounds very weird to me but I am glad you solved your issue >>> (temporary at least). >>> >>> C*heers, >>> ----------------------- >>> Alain Rodriguez - alain@thelastpickle.com >>> France >>> >>> The Last Pickle - Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >>> 2016-07-29 3:25 GMT+02:00 Eric Stevens : >>> >>>> Tombstones will not get removed even after gc_grace if bloom filters >>>> indicate that there is overlapping data with the tombstone's partition= in a >>>> different sstable. This is because compaction can't be certain that t= he >>>> tombstone doesn't overlap data in that other table. If you're writing= to >>>> one end of a partition key while deleting off the other end (for examp= le >>>> you've created engaged in the queue anti-pattern), your tombstones wil= l >>>> essentially never go away. >>>> >>>> >>>>> 220kb worth of tombstones doesn=E2=80=99t seem like enough to worry a= bout. >>>> >>>> >>>> Large range tombstones can occupy just a few bytes but can occlude >>>> millions of records, and have the corresponding performance impact on >>>> reads. It's really not the size of the tombstone on disk that matters= , but >>>> the number of records it occludes. >>>> >>>> You must either do a full compaction (while also not writing to the >>>> partitions being considered, and after you've forced a cluster-wide fl= ush, >>>> and after the tombstones are gc_grace old, and assuming size tiered an= d not >>>> leveled compaction) to get rid of those tombstones, or probably easier= is >>>> to do something similar to sstable2json, remove the tombstones by hand= , >>>> then json2sstable and replace the offending sstable. Note that you re= ally >>>> have to be certain what you're doing here or you'll end up resurrectin= g >>>> deleted records. >>>> >>>> If these all sound like bad options, it's because they are, and you >>>> don't have a lot of options without changing your schema to eventually= stop >>>> writing to (and especially reading from) partitions which you also do >>>> deletes on. https://issues.apache.org/jira/browse/CASSANDRA-7019 prop= oses >>>> to offer a better alternative, but it's still in progress. >>>> >>>> Shameless plug, I'm talking about my company's alternative to >>>> tombstones and TTLs at this year's Cassandra Summit: >>>> http://myeventagenda.com/sessions/1CBFC920-807D-41C1-942C-8D1A7C10F4FA= /5/5#sessionID=3D165 >>>> >>>> >>>> On Thu, Jul 28, 2016 at 11:07 AM sai krishnam raju potturi < >>>> pskraju88@gmail.com> wrote: >>>> >>>>> thanks a lot Alain. That was really great info. >>>>> >>>>> The issues here was that tombstones were not in the SSTable, but >>>>> rather in the Memtable. We had to a nodetool flush, and run a nodetoo= l >>>>> compact to get rid of the tombstones, a million of them. The size of = the >>>>> largest SSTable was actually 48MB. >>>>> >>>>> This link was helpful in getting the count of tombstones in a sstable= , >>>>> which was 0 in our case. >>>>> https://gist.github.com/JensRantil/063b7c56ca4a8dfe1c50 >>>>> >>>>> The application team did not have a good model. They are working >>>>> on a new datamodel. >>>>> >>>>> thanks >>>>> >>>>> On Wed, Jul 27, 2016 at 7:17 PM, Alain RODRIGUEZ >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I just released a detailed post about tombstones today that might be >>>>>> of some interest for you: >>>>>> http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstone= s.html >>>>>> >>>>>> 220kb worth of tombstones doesn=E2=80=99t seem like enough to worry = about. >>>>>> >>>>>> >>>>>> +1 >>>>>> >>>>>> I believe you might be missing some other bigger SSTable having a lo= t >>>>>> of tombstones as well. Finding the biggest sstable and reading the >>>>>> tombstone ratio from there might be more relevant. >>>>>> >>>>>> You also should give a try to: "unchecked_tombstone_compaction" set >>>>>> to true rather than tuning other options so aggressively. The "singl= e >>>>>> SSTable compaction" section of my post might help you on this issue: >>>>>> http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstone= s.html#single-sstable-compaction >>>>>> >>>>>> Other thoughts: >>>>>> >>>>>> Also if you use TTLs and timeseries, using TWCS instead of STCS coul= d >>>>>> be more efficient evicting tombstones. >>>>>> >>>>>> we have a columnfamily that has around 1000 rows, with one row is >>>>>>> really huge (million columns) >>>>>> >>>>>> >>>>>> I am sorry to say that this model does not look that great. >>>>>> Imbalances might become an issue as a few nodes will handle a lot mo= re load >>>>>> than the rest of the nodes. Also even if this is getting improved in= newer >>>>>> versions of Cassandra, wide rows are something you want to avoid whi= le >>>>>> using 2.0.14 (which is no longer supported for about a year now). I = know it >>>>>> is not always easy and never the good time, but maybe should you con= sider >>>>>> upgrading both your model and your version of Cassandra (regardless = of the >>>>>> fact you manage to solve this issue or not with >>>>>> "unchecked_tombstone_compaction"). >>>>>> >>>>>> Good luck, >>>>>> >>>>>> C*heers, >>>>>> ----------------------- >>>>>> Alain Rodriguez - alain@thelastpickle.com >>>>>> France >>>>>> >>>>>> The Last Pickle - Apache Cassandra Consulting >>>>>> http://www.thelastpickle.com >>>>>> >>>>>> 2016-07-28 0:00 GMT+02:00 sai krishnam raju potturi < >>>>>> pskraju88@gmail.com>: >>>>>> >>>>>>> The read queries are continuously failing though because of the >>>>>>> tombstones. "Request did not complete within rpc_timeout." >>>>>>> >>>>>>> thanks >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 27, 2016 at 5:51 PM, Jeff Jirsa < >>>>>>> jeff.jirsa@crowdstrike.com> wrote: >>>>>>> >>>>>>>> 220kb worth of tombstones doesn=E2=80=99t seem like enough to worr= y about. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *From: *sai krishnam raju potturi >>>>>>>> *Reply-To: *"user@cassandra.apache.org" >>>>>>>> *Date: *Wednesday, July 27, 2016 at 2:43 PM >>>>>>>> *To: *Cassandra Users >>>>>>>> *Subject: *Re: Re : Purging tombstones from a particular row in >>>>>>>> SSTable >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> and also the sstable size in question is like 220 kb in size. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 27, 2016 at 5:41 PM, sai krishnam raju potturi < >>>>>>>> pskraju88@gmail.com> wrote: >>>>>>>> >>>>>>>> it's set to 1800 Vinay. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> bloom_filter_fp_chance=3D0.010000 AND >>>>>>>> >>>>>>>> caching=3D'KEYS_ONLY' AND >>>>>>>> >>>>>>>> comment=3D'' AND >>>>>>>> >>>>>>>> dclocal_read_repair_chance=3D0.100000 AND >>>>>>>> >>>>>>>> gc_grace_seconds=3D1800 AND >>>>>>>> >>>>>>>> index_interval=3D128 AND >>>>>>>> >>>>>>>> read_repair_chance=3D0.000000 AND >>>>>>>> >>>>>>>> replicate_on_write=3D'true' AND >>>>>>>> >>>>>>>> populate_io_cache_on_flush=3D'false' AND >>>>>>>> >>>>>>>> default_time_to_live=3D0 AND >>>>>>>> >>>>>>>> speculative_retry=3D'99.0PERCENTILE' AND >>>>>>>> >>>>>>>> memtable_flush_period_in_ms=3D0 AND >>>>>>>> >>>>>>>> compaction=3D{'min_sstable_size': '1024', 'tombstone_threshold': >>>>>>>> '0.01', 'tombstone_compaction_interval': '1800', 'class': >>>>>>>> 'SizeTieredCompactionStrategy'} AND >>>>>>>> >>>>>>>> compression=3D{'sstable_compression': 'LZ4Compressor'}; >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 27, 2016 at 5:34 PM, Vinay Kumar Chella < >>>>>>>> vinaykumarcse@gmail.com> wrote: >>>>>>>> >>>>>>>> What is your GC_grace_seconds set to? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 27, 2016 at 1:13 PM, sai krishnam raju potturi < >>>>>>>> pskraju88@gmail.com> wrote: >>>>>>>> >>>>>>>> thanks Vinay and DuyHai. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> we are using verison 2.0.14. I did "user defined compaction" >>>>>>>> following the instructions in the below link, The tombstones still= persist >>>>>>>> even after that. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> https://gist.github.com/jeromatron/e238e5795b3e79866b83 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Also, we changed the tombstone_compaction_interval : 1800 >>>>>>>> and tombstone_threshold : 0.1, but it did not help. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 27, 2016 at 4:05 PM, DuyHai Doan >>>>>>>> wrote: >>>>>>>> >>>>>>>> This feature is also exposed directly in nodetool from version >>>>>>>> Cassandra 3.4 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> nodetool compact --user-defined >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 27, 2016 at 9:58 PM, Vinay Chella >>>>>>>> wrote: >>>>>>>> >>>>>>>> You can run file level compaction using JMX to get rid of >>>>>>>> tombstones in one SSTable. Ensure you set GC_Grace_seconds such th= at >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> current time >=3D deletion(tombstone time)+ GC_Grace_seconds >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> File level compaction >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> /usr/bin/java -jar cmdline-jmxclient-0.10.3.jar - localhost: >>>>>>>> >>>>>>>> =E2=80=8B{=E2=80=8B >>>>>>>> >>>>>>>> =E2=80=8Bport} >>>>>>>> >>>>>>>> org.apache.cassandra.db:type=3DCompactionManager forceUserDefined= Compaction=3D"'${KEYSPACE}','${ >>>>>>>> >>>>>>>> =E2=80=8BSSTABLEFILENAME >>>>>>>> >>>>>>>> }'"" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 27, 2016 at 11:59 AM, sai krishnam raju potturi < >>>>>>>> pskraju88@gmail.com> wrote: >>>>>>>> >>>>>>>> hi; >>>>>>>> >>>>>>>> we have a columnfamily that has around 1000 rows, with one row i= s >>>>>>>> really huge (million columns). 95% of the row contains tombstones.= Since >>>>>>>> there exists just one SSTable , there is going to be no compaction= kicked >>>>>>>> in. Any way we can get rid of the tombstones in that row? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Userdefined compaction nor nodetool compact had no effect. Any >>>>>>>> ideas folks? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>> > --001a113d35d61c4b900538cf1b76 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I haven't tested that specifically, but I haven't = bumped into any particular optimization that allows it to skip reading an s= stable where the entire relevant partition has been row-tombstoned.=C2=A0 I= t's possible that something like that could happen by examining min/max= timestamps on sstables, and not reading from any sstable with a partition-= level tombstone where the max timestamp is less than the timestamp of the p= artition tombstone.=C2=A0 However that presumes that it can have read the t= ombstones from each sstable before it read the occluded data, which I don&#= 39;t think is likely.

Such an optimization could be ther= e, but I haven't noticed it if it is, though I'm certainly not an e= xpert (more of a well informed novice).=C2=A0 If someone wants to set me st= raight on this point I'd love to know about it.

On Fri, Jul 29, 2016 at 2:37 PM DuyHai Do= an <doanduyhai@gmail.com>= wrote:
@Eric
=
Very interesting example. But then what is the case of row (= should I say partition ?) tombstones ?

Suppose tha= t in your example, I issued a=C2=A0DELETE FROM foo WHERE pk=3D'a'

=
With the same SELECT statement than before, would C* be clever e= nough to skip reading at all the whole partition (let's limit the examp= le to a single SSTable) ?

On Fri, Jul 29, 2016 at 7:00 PM, Eric Stevens <mightye= @gmail.com> wrote:
>=C2=A0Sai was= describing a timeout, not a failure due to the 100 K tombstone limit from = cassandra.yaml. But I still might be missing things about tombstones.

The trouble with tombstones is not the tombstones them= selves, rather it's that Cassandra has a lot of deleted data to read th= rough in sstables in order to satisfy a query.=C2=A0 Although if you range = constrain your cluster key in your query, the read path can optimize that r= ead to start somewhere near the correct head of your selected data, that is= _not_ true for tombstoned data. =C2=A0

Consider t= his exercise:
CREATE TABLE foo (<= /div>
=C2=A0 pk text,
=C2=A0 ck int,
= =C2=A0 PRIMARY KEY ((pk), ck)
)
INSERT INTO foo (pk,ck) VALUES (&#= 39;a', 1)
...
INSERT INTO foo (pk,ck) VALUES ('a', 100= 000)

$ nodetool flush

<= div>DELETE FROM foo WHERE pk=3D'a' AND ck = < 99999

We've now written a single &= quot;tiny" (bytes-wise) range tombstone.

Now = try to select from that table:
SELECT * = FROM foo WHERE ck > 50000 LIMIT 1
pk | ck=C2=A0
-- | ------
a =C2=A0| 100000
This has to read from the first sstable, skipping over 49999 r= ecords before it can locate the first non-tombstoned cell.

The problem isn't the size of the tombstone, tombstones themse= lves are cheaper (again, bytes-wise) than standard columns because they don= 't involve any value for the cell.=C2=A0 The problem is that the read p= ath cannot anticipate in advance what cells are going to be occluded by the= tombstone, and in order to satisfy the query it needs to read then discard= a large number of deleted cells.

The reason the t= hresholds exist in cassandra.yaml is to help guide users away from performa= nce anti-patterns that come from selects which include a large number of to= mbstoned cells.

On Thu, Jul 28, 2016 at 11:08 PM Alain RODRIGUEZ <arodrime@gmail.com> wrot= e:
Hi,

@Eric

Large range tombstones can occupy = just a few bytes but can occlude millions of records, and have the correspo= nding performance impact on reads.=C2=A0 It's really not the size of th= e tombstone on disk that matters, but the number of records it occludes.

Sai = was describing a timeout, not a failure due to the 100 K tombstone limit fr= om cassandra.yaml. But I still might be missing things about tombstones.

The read queries are continuously failing though= because of the tombstones. "Request did not complete within rpc_timeo= ut."

So that is what looks weird to me. Reading 220 kb, even holding= tombstone should probably not take that long... Or am I wrong or missing s= omething?

Your talk looks like cool stuff :-= ).

@Sai

The issues here was that= tombstones were not in the SSTable, but rather in the Memtable

This sounds weird to me = as well, knowing that memory is faster than disk and that memtables are mut= able data (so less stuff to read from there). Flushing might have triggered= some compaction, removing tombstones though.

This= still sounds very weird to me but I am glad you solved your issue (tempora= ry at least).

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra C= onsulting

2016-07-29 3:25 GMT+02:00 Eric Ste= vens <mightye@gmail.com>:
=
Tombstones will not get removed even after gc_grace if blo= om filters indicate that there is overlapping data with the tombstone's= partition in a different sstable.=C2=A0 This is because compaction can'= ;t be certain that the tombstone doesn't overlap data in that other tab= le.=C2=A0 If you're writing to one end of a partition key while deletin= g off the other end (for example you've created engaged in the queue an= ti-pattern), your tombstones will essentially never go away.
=C2=A0
220kb worth = of tombstones doesn=E2=80=99t seem like enough to worry about.
Large range tombstones can occupy just a few bytes b= ut can occlude millions of records, and have the corresponding performance = impact on reads.=C2=A0 It's really not the size of the tombstone on dis= k that matters, but the number of records it occludes.

=
You must either do a full compaction (while also not writing to the pa= rtitions being considered, and after you've forced a cluster-wide flush= , and after the tombstones are gc_grace old, and assuming size tiered and n= ot leveled compaction) to get rid of those tombstones, or probably easier i= s to do something similar to sstable2json, remove the tombstones by hand, t= hen json2sstable and replace the offending sstable.=C2=A0 Note that you rea= lly have to be certain what you're doing here or you'll end up resu= rrecting deleted records.

If these all sound like = bad options, it's because they are, and you don't have a lot of opt= ions without changing your schema to eventually stop writing to (and especi= ally reading from) partitions which you also do deletes on. =C2=A0https://issues.apache.org/jira/browse/CASSANDRA-7019=C2=A0proposes to = offer a better alternative, but it's still in progress. =C2=A0

Shameless plug, I'm talking about my company's alt= ernative to tombstones and TTLs at this year's Cassandra Summit:=C2=A0<= a href=3D"http://myeventagenda.com/sessions/1CBFC920-807D-41C1-942C-8D1A7C1= 0F4FA/5/5#sessionID=3D165" target=3D"_blank">http://myeventagenda.com/sessi= ons/1CBFC920-807D-41C1-942C-8D1A7C10F4FA/5/5#sessionID=3D165
=

O= n Thu, Jul 28, 2016 at 11:07 AM sai krishnam raju potturi <pskraju88@gmail.com> wro= te:
thanks a lot Alain. That was really great info.=C2=A0

The issues her= e was that tombstones were not in the SSTable, but rather in the Memtable. = We had to a nodetool flush, and run a nodetool compact to get rid of the to= mbstones, a million of them. The size of the largest SSTable was actually 4= 8MB.

T= his link was helpful in getting the count of tombstones in a sstable, which= was 0 in our case.=C2=A0
https:= //gist.github.com/JensRantil/063b7c56ca4a8dfe1c50

=C2=A0 =C2=A0 The app= lication team did not have a good model. They are working on a new datamode= l.

tha= nks

On Wed, Jul 27, 2016 at 7:17 PM, Alain RODRIGUEZ <arodri= me@gmail.com> wrote:
Hi,= =C2=A0

I just released a detailed post about tombstones = today that might be of some interest for you: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html=

220kb worth of tombstones doesn=E2=80= =99t seem like enough to worry about.

+1

I believe you might be missing some ot= her bigger SSTable having a lot of tombstones as well. Finding the biggest = sstable and reading the tombstone ratio from there might be more relevant.<= /div>

You also should give a try to: "uncheck= ed_tombstone_compaction" set to true rather than tuning other options = so aggressively. The "single SSTable compaction" section of my po= st=C2=A0might help you on this issue: http://thelastpickle.com/blog/2016/07/27/about-deletes-a= nd-tombstones.html#single-sstable-compaction

O= ther thoughts:

Also if you use TTLs and times= eries, using TWCS instead of STCS could be more efficient evicting tombston= es.

we have a columnfamily that has around 1000 rows, with one row i= s really huge (million columns)

I am sorry to say that this model does not look that great. Imbalances m= ight become an issue as a few nodes will handle a lot more load than the re= st of the nodes. Also even if this is getting improved in newer versions of= Cassandra, wide rows are something you want to avoid while using 2.0.14 (w= hich is no longer supported for about a year now). I know it is not always = easy and never the good time, but maybe should you consider upgrading both = your model and your version of Cassandra (regardless of the fact you manage= to solve this issue or not with "unchecked_tombstone_compaction"= ).

Good luck,


2016-07-28 0:00 GMT+= 02:00 sai krishnam raju potturi <pskraju88@gmail.com>:
=
The read queries are continuously failing= though because of the tombstones. "Request did not complete within rp= c_timeout."

thanks

<= div>

On Wed, Jul 2= 7, 2016 at 5:51 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com>= ; wrote:
<= div>

220kb worth of to= mbstones doesn=E2=80=99t seem like enough to worry about.

=C2=A0<= u>

<= /u>=C2=A0

From: = sai krishnam raju pottu= ri <pskraju88@g= mail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.a= pache.org>
Date: Wednesday, July 27, 2016 at 2:43 PM
To: Cassandra Users <user@cassandra.apache.org>
Subject: Re: R= e : Purging tombstones from a particular row in SSTable

=C2=A0

<= p>and also the sstable size in question is like 220 kb in size.

= =C2=A0

thanks

=C2=A0

=C2=A0

On We= d, Jul 27, 2016 at 5:41 PM, sai krishnam raju potturi <pskraju88@gmail.com> wrote:<= u>

it's set to 18= 00 Vinay.

=C2=A0

=C2=A0bloom_filter_fp_chance=3D0.010000 AND

=C2=A0 caching=3D'KEYS_ONLY' AND

=C2=A0 comment=3D'' AND

=C2=A0 dclo= cal_read_repair_chance=3D0.100000 AND

=C2=A0= gc_grace_seconds=3D1800= AND

=C2=A0 index_interval=3D128 AND<= u>

=C2=A0 read_repair_chance=3D0.000000 AND<= /u>

=C2=A0 replicate_on_write=3D'true' AND<= u>

=C2=A0 populate_io_cache_on_flush=3D'false'= AND

=C2=A0 default_time_to_live=3D0 AND<= /u>

=C2=A0 speculative_retry=3D'99.0PERCENTILE&= #39; AND

=C2=A0 memtable_flush_period_in_ms= =3D0 AND

=C2=A0 compaction=3D{'min_sstab= le_size': '1024', 'tombstone_threshold': '0.01'= , 'tombstone_compaction_interval': '1800', 'class':= 'SizeTieredCompactionStrategy'} AND

=C2=A0 compression=3D{'sstable_compression': 'LZ4Compressor= 9;};

=C2=A0

thanks

=C2=A0

=C2=A0

On Wed, Jul 27, 2016 = at 5:34 PM, Vinay Kumar Chella <vinaykumarcse@gmail.com> wrote:

What is your GC_grace_seconds set to?

=C2=A0

On Wed, Jul 27, 2016 at = 1:13 PM, sai krishnam raju potturi <pskraju88@gmail.com> wrote:

thanks Vinay and DuyHai.

=C2=A0

=C2=A0 =C2=A0 we are us= ing verison 2.0.14. I did "user defined compaction" following the= instructions in the below link, The tombstones still persist even after th= at.

=C2=A0

https://gist= .github.com/jeromatron/e238e5795b3e79866b83

=C2=A0

Also, we changed the=C2=A0tombsto= ne_compaction_interval : 1800 and=C2=A0tombstone_threshold : 0.1, but it di= d not help.

=C2=A0

<= /div>

thanks

= =C2=A0

=C2=A0

=C2=A0

On Wed, Jul 27, 2016 at 4:05 PM, DuyHai Doan <= doanduyhai@gmail.= com> wrote:

This feature is also exposed directly in nodetool from version Cassandra 3= .4

=C2=A0

nodetool= compact --user-defined=C2=A0<SSTable file>

=C2=A0

On Wed, Jul 27, 2016 = at 9:58 PM, Vinay Chella <vchella@netflix.com> wrote:

You can= run file level compaction using JMX to get rid of tombstones in one SSTabl= e. Ensure you set GC_Grace_seconds such that=C2=A0

=

=C2=A0=

current time >=3D deletion(= tombstone time)+ GC_Grace_seconds=C2=A0

=

=C2=A0

<= div>

File level compaction

=C2=A0

/usr/bin/java -jar cmdline-jmxclient-0.10.3.jar - l=
ocalhost:
=E2=80=8B{=E2=80=8B
=E2=80=8Bport}
 org.apache.cassandr=
a.db:type=3DCompactionManager forceUserDefinedCompaction=3D"'${KEY=
SPACE}','${
=E2=80=8BSSTABLEFILENAME
}'""<=
u>

=C2=A0


=C2=A0

=C2=A0

On = Wed, Jul 27, 2016 at 11:59 AM, sai krishnam raju potturi <pskraju88@gmail.com> wrot= e:

hi; <= /u>

=C2=A0 we have a columnfamily that has around 1000 rows, wit= h one row is really huge (million columns). 95% of the row contains tombsto= nes. Since there exists just one SSTable , there is going to be no compacti= on kicked in. Any way we can get rid of the tombstones in that row?<= u>

=C2=A0

Userdefined c= ompaction nor nodetool compact had no effect. Any ideas folks?

=C2=A0

thanks

=C2=A0

=C2=A0<= /u>

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0






--001a113d35d61c4b900538cf1b76--