Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F401117A11 for ; Tue, 3 Feb 2015 20:50:48 +0000 (UTC) Received: (qmail 15489 invoked by uid 500); 3 Feb 2015 20:50:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15448 invoked by uid 500); 3 Feb 2015 20:50:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15438 invoked by uid 99); 3 Feb 2015 20:50:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 20:50:30 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of arodrime@gmail.com designates 209.85.215.49 as permitted sender) Received: from [209.85.215.49] (HELO mail-la0-f49.google.com) (209.85.215.49) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 20:50:02 +0000 Received: by mail-la0-f49.google.com with SMTP id gf13so54424188lab.8 for ; Tue, 03 Feb 2015 12:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=Wbf3UUjJiYQ4FqHHYRJXmZdC0twHxEbDvJpKzT5vbLA=; b=VlLXc0dD4wk6KRLFYLt3Toz4qBlFWoEWocZAonz78Af7ZwOUHcOMQ8nbTeRS9gCKX/ ffHFJfoyzF3TcRrJqi/dpw+8ROIRQ9gZM8KOXcm0SMzCdppUGpGVdev2Ph27sLKtziT+ PZe20t1sd89HS36eGURftVwU76c5ZfHbtmg6LzaUPa99PXG+kyPNShu7xTNRAlfJ2DWO 86R36Qiue2HG1uZjUUzyCY9MVzZ0TqSYrz4tedoPjmj2qgLubtjWNbd87OThjhE6Qv0y VcD0svDBjHGNc3aXMPV75pcJxZYPfo6vMyPKltA6jqDpQRWdPrVCeNPzKQGiLuBEvdAS QRIQ== X-Received: by 10.152.36.162 with SMTP id r2mr14017692laj.9.1422996510927; Tue, 03 Feb 2015 12:48:30 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.33.168 with HTTP; Tue, 3 Feb 2015 12:48:10 -0800 (PST) In-Reply-To: <1E807EB9EA831F4184920D0403D4CD6D014F6D24@exchange2010.dimcap.corp> References: <1E807EB9EA831F4184920D0403D4CD6D014EE1C7@exchange2010.dimcap.corp> <1E807EB9EA831F4184920D0403D4CD6D014F4A3A@exchange2010.dimcap.corp> <045D8FD556C73347A47F956EE65F8220185B4DA4@S11MAILD013N2.sh11.lan> <1E807EB9EA831F4184920D0403D4CD6D014F6D24@exchange2010.dimcap.corp> From: Alain RODRIGUEZ Date: Tue, 3 Feb 2015 21:48:10 +0100 Message-ID: Subject: Re: Tombstone gc after gc grace seconds To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0158b79c8cf0f6050e353196 X-Virus-Checked: Checked by ClamAV on apache.org --089e0158b79c8cf0f6050e353196 Content-Type: text/plain; charset=ISO-8859-1 Hi, thanks for sharing your tests ! Though, how did you inserted the data ? Did you try adding columns in an atomic and random order, with a small memtable size to achieve a big sharding (normal in time series use case) ? I think performing ./md_test against this set of data would be interesting, taking care to shard enough to have parts of each key on each SSTable, or in many at least. This would measure the effectiveness of this parameter in a "normal" time series workflow (which is a standard use case of C*). By the way, this might be done on both LCS and STCS and see how both behave. Thanks again, it is interesting to hear about tests like yours ! C*heers Alain 2015-01-30 17:32 GMT+01:00 Ravi Agrawal : > I did a small test. I wrote data to 4 different column family. 30MB of > data. > > 256 rowkeys and 100K columns on an average. > > And then deleted all data from all of them. > > > > 1. Md_normal - created using default compaction parameters and Gc > Grace seconds was 5 seconds. Data was written and then deleted. Compaction > was ran using "nodetool compact keyspace columnfamily" - I see full disk > data, but cannot query columns(since data was deleted consistent behavior) > and cannot query rows in cqlsh. Hits timeout. > > 2. Md_test - created using following compact parameters - > "compaction={'tombstone_threshold': '0.000001', 'class': > 'SizeTieredCompactionStrategy'}" and Gc Grace seconds was 5 seconds. > Disksize is reduced, and am able to query rows which return 0. > > 3. Md_test2 - created using following compact parameters - > "compaction={'tombstone_threshold': '0.0', 'class': > 'SizeTieredCompactionStrategy'}". Disksize is reduced, not able to query > rows using cqlsh. Hits timeout. > > 4. Md_forcecompact - created using compaction parameters > "compaction={'unchecked_tombstone_compaction': 'true', 'class': > 'SizeTieredCompactionStrategy'}" and Gc Grace seconds was 5 seconds. Data > was written and then deleted. I see full disk data, but cannot query any > data using mddbreader and cannot query rows in cqlsh. Hits timeout. > > > > Next day sizes were - > > 30M ./md_forcecompact > > 4.0K ./md_test > > 304K ./md_test2 > > 30M ./md_normal > > > > Feel of the data that we have is - > > 8000 rowkeys per day and columns are added throughout the day. 300K > columns on an average per rowKey. > > > > > > > > *From:* Alain RODRIGUEZ [mailto:arodrime@gmail.com] > *Sent:* Friday, January 30, 2015 4:26 AM > > *To:* user@cassandra.apache.org > *Subject:* Re: Tombstone gc after gc grace seconds > > > > The point is that all the "parts" or "fragments" of the row need to be in > the SSTables implied in the compaction for C* to be able to evict the row > effectively. > > > > My understanding of those parameters is that they will trigger a > compaction on the SSTable that exceed this ratio. This will work properly > if you never "update" a row (by modifying a value or adding a column). If > your workflow is something like "Write once per partition key", this > parameter will do the job. > > > > If you have fragments, you might trigger this compaction for nothing. In > the case of frequently updated rows (like when using wide rows / time > series) your only way to get rid of tombstone is a major compaction. > > > > That's how I understand this. > > > > Hope this help, > > > > C*heers, > > > > Alain > > > > 2015-01-30 1:29 GMT+01:00 Mohammed Guller : > > Ravi - > > > > It may help. > > > > What version are you running? Do you know if minor compaction is getting > triggered at all? One way to check would be see how many sstables the data > directory has. > > > > Mohammed > > > > *From:* Ravi Agrawal [mailto:ragrawal@clearpoolgroup.com] > *Sent:* Thursday, January 29, 2015 1:29 PM > *To:* user@cassandra.apache.org > *Subject:* RE: Tombstone gc after gc grace seconds > > > > Hi, > > I saw there are 2 more interesting parameters - > > a. tombstone_threshold - A ratio of garbage-collectable tombstones > to all contained columns, which if exceeded by the SSTable triggers > compaction (with no other SSTables) for the purpose of purging the > tombstones. Default value - 0.2 > > b. unchecked_tombstone_compaction - True enables more aggressive > than normal tombstone compactions. A single SSTable tombstone compaction > runs without checking the likelihood of success. Cassandra 2.0.9 and later. > > Could I use these to get what I want? > > Problem I am encountering is even long after gc_grace_seconds I see no > reduction in disk space until I run compaction manually. I was thinking to > make tombstone threshold close to 0 and unchecked compaction set to true. > > Also we are not running nodetool repair on weekly basis as of now. > > > > *From:* Eric Stevens [mailto:mightye@gmail.com ] > *Sent:* Monday, January 26, 2015 12:11 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Tombstone gc after gc grace seconds > > > > My understanding is consistent with Alain's, there's no way to force a > tombstone-only compaction, your only option is major compaction. If you're > using size tiered, that comes with its own drawbacks. > > > > I wonder if there's a technical limitation that prevents introducing a > shadowed data cleanup style operation (overwritten data, including deletes, > plus tombstones past their gc grace period); or maybe even couple it > directly with cleanup since most of the work (rewriting old SSTables) would > be identical. I can't think of something off the top of my head, but it > would be so useful that it seems like there's got to be something I'm > missing. > > > > On Mon, Jan 26, 2015 at 4:15 AM, Alain RODRIGUEZ > wrote: > > I don't think that such a thing exists as SSTables are immutable. You > compact it entirely or you don't. Minor compaction will eventually evict > tombstones. If it is too slow, AFAIK, the "better" solution is a major > compaction. > > > > C*heers, > > > > Alain > > > > 2015-01-23 0:00 GMT+01:00 Ravi Agrawal : > > Hi, > > I want to trigger just tombstone compaction after gc grace seconds is > completed not nodetool compact keyspace column family. > > Anyway I can do that? > > > > Thanks > > > > > > > > > > > --089e0158b79c8cf0f6050e353196 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi, thanks for sharing your tests !

Tho= ugh, how did you inserted the data ? Did you try adding columns in an atomi= c and random order, with a small memtable size to achieve a big sharding (n= ormal in time series use case) ?

I think performin= g ./md_test against this set of data would be interesting, taking care to s= hard enough to have parts of each key on each SSTable, or in many at least.=

This would measure the effectiveness of this para= meter in a "normal" time series workflow (which is a standard use= case of C*).

By the way, this might be done on bo= th LCS and STCS=  and see how both behave.

Thanks again, i= t is interesting to hear about tests like yours !

= C*heers

Alain

2015-01-30 17:32 GMT+01:00 Ravi Agrawal <ragrawal@clearpoolgroup.com>:

I did a small test. I wrote data to 4= different column family. 30MB of data.

256 rowkeys and 100K columns on an av= erage.

And then deleted all data from all of= them.

 

1.       Md_normal –= created using default compaction parameters and Gc Grace seconds was 5 sec= onds. Data was written and then deleted. Compaction was ran using “no= detool compact keyspace columnfamily” – I see full disk data, but cannot query columns(since data was deleted consistent= behavior) and cannot query rows in cqlsh. Hits timeout.

2.     &nbs= p; M= d_test – created using following compact parameters – “co= mpaction=3D{'tombstone_threshold': '0.000001', 'class&#= 39;: 'SizeTieredCompactionStrategy'}” and Gc Grace seconds was 5 seconds. Disksize is reduced, and am able to query rows which return= 0.

3.       Md_test2 – = created using following compact parameters – “compaction=3D{= 9;tombstone_threshold': '0.0', 'class': 'SizeTiered= CompactionStrategy'}”. Disksize is reduced, not able to query row= s using cqlsh. Hits timeout.

4.       Md_forcecompact &= ndash; created using compaction parameters “compaction=3D{'unchec= ked_tombstone_compaction': 'true', 'class': 'SizeTi= eredCompactionStrategy'}” and Gc Grace seconds was 5 seconds. Dat= a was written and then deleted. I see full disk data, but cannot query any data using md= dbreader and cannot query rows in cqlsh. Hits timeout.=

 

Next day sizes were –=

30M     ./md_forc= ecompact

4.0K   = ; ./md_test

304K    ./md_test2<= /u>

30M     ./md_norm= al

 

Feel of the data that we have is -

8000 rowkeys per day and columns are = added throughout the day. 300K columns on an average per rowKey.<= /u>

 

 

 

From: Alain RODRIGUEZ [mailto:arodrime@gmail.com]
Sent: Friday, January 30, 2015 4:26 AM


To: u= ser@cassandra.apache.org
Subject: Re: Tombstone gc after gc grace seconds
=

 

The point is that all the "parts" or "= ;fragments" of the row need to be in the SSTables implied in the compa= ction for C* to be able to evict the row effectively.

 

My understanding of those parameters is that they wi= ll trigger a compaction on the SSTable that exceed this ratio. This will wo= rk properly if you never "update" a row (by modifying a value or = adding a column). If your workflow is something like "Write once per partition key", this parameter will do the = job.

 

If you have fragments, you might trigger this compac= tion for nothing. In the case of frequently updated rows (like when using w= ide rows / time series) your only way to get rid of tombstone is a major co= mpaction.

 

That's how I understand this.

 

Hope this help,

 

C*heers,

 

Alain

 

2015-01-30 1:29 GMT+01:00 Mohammed Guller <mohammed@glassbeam.co= m>:

Ravi –

 

It may help.

 

What version are you running? Do you = know if minor compaction is getting triggered at all? One way to check would be see how many sstables the data directory has. =

 

Mohammed

 

From: Ravi Agrawal [mailto:ragrawal@clearpoolgro= up.com]
Sent: Thursday, January 29, 2015 1:29 PM
To: u= ser@cassandra.apache.org
Subject: RE: Tombstone gc after gc grace seconds

 

Hi,

I saw there are 2 more interesting paramete= rs –

a. &nbs= p;     tombstone_threshold -= A ratio of garbage-collectable tombstones to all contained columns, which = if exceeded by the SSTable triggers compaction (with no other SSTables) for t= he purpose of purging the tombstones. Default value – 0.2

b. &nbs= p;    unchecked_tombstone_compaction - True enables more aggressive than normal tombstone compactions. A sin= gle SSTable tombstone compaction runs without checking the likelihood of succe= ss. Cassandra 2.0.9 and later.

Could I use these to get what I want?

Problem I am encountering is even long afte= r gc_grace_seconds I see no reduction in disk space until I run compaction = manually. I was thinking to make tombstone threshold close to 0 and unchecked compaction set to true.

Also we are not running nodetool repair on = weekly basis as of now.

 

From: Eric Stevens [mailto:mightye@gmail.com]
Sent: Monday, January 26, 2015 12:11 PM
To: u= ser@cassandra.apache.org
Subject: Re: Tombstone gc after gc grace seconds

 

My understanding is consistent with Alain's, the= re's no way to force a tombstone-only compaction, your only option is m= ajor compaction.  If you're using size tiered, that comes with its own drawbacks.

 

I wonder if there's a technical limitation that = prevents introducing a shadowed data cleanup style operation (overwritten d= ata, including deletes, plus tombstones past their gc grace period); or maybe even couple it directly with cleanup since most of= the work (rewriting old SSTables) would be identical.  I can't th= ink of something off the top of my head, but it would be so useful that it = seems like there's got to be something I'm missing. 

 

On Mon, Jan 26, 2015 at 4:15 AM, Alain RODRIGUEZ <= ;arodrime@gmail.com= > wrote:

I don't think that such a thing exists as SSTabl= es are immutable. You compact it entirely or you don't. Minor compactio= n will eventually evict tombstones. If it is too slow, AFAIK, the "better" solution is a major compaction.

 

C*heers,

 

Alain

 

2015-01-23 0:00 GMT+01:00 Ravi Agrawal <ragrawal@clearpoo= lgroup.com>:

Hi,

I want to trigger just tombstone compaction after gc= grace seconds is completed not nodetool compact keyspace column family.=

Anyway I can do that?

 

Thanks

 

 

 

 

 


--089e0158b79c8cf0f6050e353196--