Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A0C2817361 for ; Tue, 21 Apr 2015 16:11:28 +0000 (UTC) Received: (qmail 84658 invoked by uid 500); 21 Apr 2015 16:11:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84613 invoked by uid 500); 21 Apr 2015 16:11:25 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84603 invoked by uid 99); 21 Apr 2015 16:11:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Apr 2015 16:11:25 +0000 X-ASF-Spam-Status: No, hits=-5.8 required=5.0 tests=ENV_AND_HDR_SPF_MATCH,HTML_MESSAGE,SPF_PASS,USER_IN_DEF_SPF_WL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which is an MX secondary for user@cassandra.apache.org) Received: from [54.76.25.247] (HELO mx1-eu-west.apache.org) (54.76.25.247) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Apr 2015 16:10:59 +0000 Received: from mail-wg0-f47.google.com (mail-wg0-f47.google.com [74.125.82.47]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id B485125F74 for ; Tue, 21 Apr 2015 16:09:13 +0000 (UTC) Received: by wgso17 with SMTP id o17so219141558wgs.1 for ; Tue, 21 Apr 2015 09:09:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=vxQBNb2UiQfAjJeaWuN/96X3347dQdH14NovMN8rOQM=; b=KrcM5CnDfYDQtHOrzc5nJChm7TiFzfodCmx5a8ZSQxyDWa0BU7Puefl1mhdpC+66Zt 2TDQYz4CRLbMwbsO+Y7p8rUm0WTEbfA6BOriXnjRbp1/O+AbNaI8A8hZrsRW+sFfVEMV lBJohhfQyhR9TSePcTdQu3aZtLBMhr4IlPNX7PsdywHys7jr8JgnzdWhAf7wZpt1/vln BD+Hrki7yhsrntMUCImZQdSG9jPtKIGjPVrla4669X9mAxWE5b5z9SzvTX8n8fAwrfb5 tnQ11pQT/QcpAySfFh4RP+L2om+hLQQE65VEamgHXAZf1OD/3mbhvQWj8dm6UEcfXD5e EDAw== X-Gm-Message-State: ALoCoQktQgwln1nAB1gwvfDxJr1x6iYRxYX3rd8NfCxx+2xu99lb2l3v9sBjRnZdYzkEfjB/uYTp MIME-Version: 1.0 X-Received: by 10.180.82.135 with SMTP id i7mr37314869wiy.68.1429632553324; Tue, 21 Apr 2015 09:09:13 -0700 (PDT) Received: by 10.28.151.144 with HTTP; Tue, 21 Apr 2015 09:09:13 -0700 (PDT) In-Reply-To: References: Date: Tue, 21 Apr 2015 12:09:13 -0400 Message-ID: Subject: Re: Cassandra tombstones being created by updating rows with TTL's From: "Laing, Michael" To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=f46d04428da280314605143e445a X-Virus-Checked: Checked by ClamAV on apache.org --f46d04428da280314605143e445a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Discussions previously on the list show why this is not a problem in much more detail. If something changes in your cluster: node down, new node, etc - you run repair for sure. We also run periodic repairs prophylactically. But if you never delete and always ttl by the same amount, you do not have to worry about zombie data being resurrected - the main reason for running repair within gc_grace_seconds. On Tue, Apr 21, 2015 at 11:49 AM, Walsh, Stephen wrote: > Maybe thanks Michael, > > I will give these setting a go, > > How do you do you periodic node-tool repairs in the situation, for what I > read we need to start doing this also. > > > > https://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair > > > > > > *From:* Laing, Michael [mailto:michael.laing@nytimes.com] > *Sent:* 21 April 2015 16:26 > *To:* user@cassandra.apache.org > *Subject:* Re: Cassandra tombstones being created by updating rows with > TTL's > > > > If you never delete except by ttl, and always write with the same ttl (or > monotonically increasing), you can set gc_grace_seconds to 0. > > > > That's what we do. There have been discussions on the list over the last > few years re this topic. > > > > ml > > > > On Tue, Apr 21, 2015 at 11:14 AM, Walsh, Stephen > wrote: > > We were chatting to Jon Haddena about a week ago about our tombstone > issue using Cassandra 2.0.14 > > To Summarize > > > > We have a 3 node cluster with replication-factor=3D3 and compaction =3D > SizeTiered > > We use 1 keyspace with 1 table > > Each row have about 40 columns > > Each row has a TTL of 10 seconds > > > > We insert about 500 rows per second in a prepared batch** (about 3mb in > network overhead) > > We query the entire table once per second > > > > **This is too enable consistent data, E.G batch in transactional, so we > get all queried data from one insert and not a mix of 2 or more. > > > > > > Seems every second we insert, the rows are never deleted by the TTL, or s= o > we thought. > > After some time we got this message on the query side > > > > > > ####################################### > > ERROR [ReadStage:91] 2015-04-21 12:27:03,902 SliceQueryFilter.java (line > 206) Scanned over 100000 tombstones in keyspace.table; query aborted (see > tombstone_failure_threshold) > > ERROR [ReadStage:91] 2015-04-21 12:27:03,931 CassandraDaemon.java (line > 199) Exception in thread Thread[ReadStage:91,5,main] > > java.lang.RuntimeException: > org.apache.cassandra.db.filter.TombstoneOverwhelmingException > > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StoragePr= oxy.java:2008) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException > > ####################################### > > > > > > So we know tombstones are infact being created. > > Solution was to change the table schema and set gc_grace_seconds to run > every 60 seconds. > > This worked for 20 seconds, then we saw this > > > > > > ####################################### > > Read 500 live and 30000 tombstoned cells in keyspace.table (see > tombstone_warn_threshold). 10000 columns was requested, slices=3D[-], > delInfo=3D{deletedAt=3D-9223372036854775808, localDeletion=3D2147483647} > > ####################################### > > > > So every 20 seconds (500 inserts x 20 seconds =3D 10,000 tombstones) > > So now we have the gc_grace_seconds set to 10 seoncds. > > But its feels very wrong to have it at a low number, especially if we mov= e > to a larger cluster. This just wont fly. > > What are we doing wrong? > > > > We shouldn=E2=80=99t increase the tombstone threshold as that is extremel= y > dangerous. > > > > > > Best Regards > > Stephen Walsh > > > > > > > > > > > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have receiv= ed > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > This email (including any attachments) is proprietary to Aspect > Software, Inc. and may contain information that is confidential. If you > have received this message in error, please do not read, copy or forward > this message. Please notify the sender immediately, delete it from your > system and destroy any copies. You may not further disclose or distribute > this email or its attachments. > --f46d04428da280314605143e445a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Discussions previously on the list show why this is not a = problem in much more detail.

If something changes in you= r cluster: node down, new node, etc - you run repair for sure.
We also run periodic repairs prophylactically.

<= /div>
But if you never delete and always ttl by the same amount, you do= not have to worry about zombie data being resurrected - the main reason fo= r running repair within gc_grace_seconds.



On Tue, A= pr 21, 2015 at 11:49 AM, Walsh, Stephen <Stephen.Walsh@aspect.com> wrote:

Maybe thanks Michael,

I will give these setting a go,

How do you do you periodic node-tool = repairs in the situation, for what I read we need to start doing this also.=

=C2=A0

https://= wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair

=C2=A0

=C2=A0

From: = Laing, Michael [mailto:michael.laing@nytimes.com]
Sent: 21 April 2015 16:26
To: u= ser@cassandra.apache.org
Subject: Re: Cassandra tombstones being created by updating rows wit= h TTL's

=C2=A0

If you never delete except by ttl, and always write = with the same ttl (or monotonically increasing), you can set gc_grace_secon= ds to 0.

=C2=A0

That's what we do. There have been discussions o= n the list over the last few years re this topic.

=C2=A0

ml

=C2=A0

On Tue, Apr 21, 2015 at 11:14 AM, Walsh, Stephen <= ;Stephen.Wals= h@aspect.com> wrote:

We were chatting to Jon Haddena about a week ago abo= ut our tombstone issue using Cassandra 2.0.14

To Summarize

=C2=A0

We have a 3 node cluster with replication-factor=3D3= and compaction =3D SizeTiered

We use 1 keyspace with 1 table

Each row have about 40 columns

Each row has a TTL of 10 seconds

=C2=A0

We insert about 500 rows per second in a prepared ba= tch** (about 3mb in network overhead)

We query the entire table once per second<= /u>

=C2=A0

**This is too enable consistent data, E.G batch in t= ransactional, so we get all queried data from one insert and not a mix of 2= or more.

=C2=A0

=C2=A0

Seems every second we insert, the rows are never del= eted by the TTL, or so we thought.

After some time we got this message on the query sid= e

=C2=A0

=C2=A0

#######################################

ERROR [ReadStage:91] 2015-04-21 12:27:03,902 SliceQu= eryFilter.java (line 206) Scanned over 100000 tombstones in keyspace.table;= query aborted (see tombstone_failure_threshold)

ERROR [ReadStage:91] 2015-04-21 12:27:03,931 Cassand= raDaemon.java (line 199) Exception in thread Thread[ReadStage:91,5,main]=

java.lang.RuntimeException: org.apache.cassandra.db.= filter.TombstoneOverwhelmingException

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.Sto= rageProxy$DroppableRunnable.run(StorageProxy.java:2008)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolE= xecutor.runWorker(ThreadPoolExecutor.java:1142)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolE= xecutor$Worker.run(ThreadPoolExecutor.java:617)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java= :745)

Caused by: org.apache.cassandra.db.filter.TombstoneO= verwhelmingException

#######################################

=C2=A0

=C2=A0

So we know tombstones are infact being created.

Solution was to change the table schema and set gc_g= race_seconds to run every 60 seconds.

This worked for 20 seconds, then we saw this<= u>

=C2=A0

=C2=A0

#######################################

Read 500 live and 30000 tombstoned cells in keyspace= .table (see tombstone_warn_threshold). 10000 columns was requested, slices= =3D[-], delInfo=3D{deletedAt=3D-9223372036854775808, localDeletion=3D2147483647}

#######################################

=C2=A0

So every 20 seconds (500 inserts x 20 seconds =3D 10= ,000 tombstones)

So now we have the gc_grace_seconds set to 10 seoncd= s.

But its feels very wrong to have it at a low number,= especially if we move to a larger cluster. This just wont fly.

What are we doing wrong?

=C2=A0

We shouldn=E2=80=99t increase the tombstone threshol= d as that is extremely dangerous.

=C2=A0

=C2=A0

Best Regards

Stephen Walsh

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

This email (including any attachments) is proprietar= y to Aspect Software, Inc. and may contain information that is confidential= . If you have received this message in error, please do not read, copy or f= orward this message. Please notify the sender immediately, delete it from your system and destroy any copies.= You may not further disclose or distribute this email or its attachments.

=C2=A0

This email (including any attachments) is proprietary to Aspect Software, I= nc. and may contain information that is confidential. If you have received = this message in error, please do not read, copy or forward this message. Pl= ease notify the sender immediately, delete it from your system and destroy any copies. You may not further dis= close or distribute this email or its attachments.

--f46d04428da280314605143e445a--