Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9B20A9F6A for ; Sun, 5 Feb 2012 20:50:48 +0000 (UTC) Received: (qmail 12104 invoked by uid 500); 5 Feb 2012 20:50:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 12011 invoked by uid 500); 5 Feb 2012 20:50:45 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 12003 invoked by uid 99); 5 Feb 2012 20:50:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Feb 2012 20:50:45 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a57.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Feb 2012 20:50:37 +0000 Received: from homiemail-a57.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a57.g.dreamhost.com (Postfix) with ESMTP id 8D92A208060 for ; Sun, 5 Feb 2012 12:50:14 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=AHwwsxJELt vUyQcX2ofRlkSvei3/IzJ2t7xXTjkn9/ikfS+80/QW31Krycp85ztVgt+f34Nktq nsPrPHC1QMl2rnl4k9Gk7ECx+azXYUuK8xift3FkXpTUSZW3ZwbJ3Jsfp0QF4rvh X/p7ONaew/vXj7zTnmE7/2TScJM/zHfZQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=aI35xEo0gNqicmPG mFymmBpKcKQ=; b=bWL9rASrZwl8DB5Nz0wjdREVOh3LxY1Cp3N2bPZNUMkiFUCw Ey+rE5/5zTqON9HW7anw7NVgjqwKbNJDstAI1IP60B7k7hp7G8uNZ0lPn7Npzwsw cQYSXmK6SN0JMpFx7GCmNToDMSzHB9M90HHBDaUu1S++d9+/SY6+enIL/fQ= Received: from [172.16.1.3] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a57.g.dreamhost.com (Postfix) with ESMTPSA id 6D6E620805B for ; Sun, 5 Feb 2012 12:50:13 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: multipart/alternative; boundary="Apple-Mail=_48311E22-1CAC-4B8C-A583-4F613709410B" Subject: Re: Consurrent compactors Date: Mon, 6 Feb 2012 09:50:07 +1300 In-Reply-To: <2C85E14562B39345BCCAD90B8E7955C90868AB@DKEXC001.adform.com> To: user@cassandra.apache.org References: <2C85E14562B39345BCCAD90B8E7955C9085DF0@DKEXC001.adform.com> <69B2567D-6437-470F-935B-7AB882ABA6BF@thelastpickle.com> <2C85E14562B39345BCCAD90B8E7955C90868AB@DKEXC001.adform.com> Message-Id: <4C1700B1-51AA-4E94-8931-2E0095254F43@thelastpickle.com> X-Mailer: Apple Mail (2.1251.1) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_48311E22-1CAC-4B8C-A583-4F613709410B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Not sure I understand the question. Do you have an example where a CF is = not getting compacted ?=20 The compaction tasks will be processed in the order they are submitted. = If you have concurrent_compactors > 1 then the thread pool for = compactions (excluding validation compactions) will be able to process = multiple compaction tasks in parallel.=20 If you have a CF that gets a lot more traffic than other CF's it will = require more compaction. But by running concurrent_compactors > 1 = smaller CF's should still be able to get through.=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 3/02/2012, at 9:15 PM, Viktor Jevdokimov wrote: > My concern is not anout cleanup, but about supposed =84tendency of = small sstables to accumulate during a single long running compactions=93. = When next task is for the same column family as currently long-running = compaction, other column families compactions are freezed and = concurrent_compactors > 1setting just not working. > =20 > =20 > Best regards/ Pagarbiai > =20 > Viktor Jevdokimov > Senior Developer > =20 > Email: Viktor.Jevdokimov@adform.com > Phone: +370 5 212 3063. Fax: +370 5 261 0453 > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania > =20 > =20 > >=20 > > Follow: >=20 > > Visit our blog >=20 > Disclaimer: The information contained in this message and attachments = is intended solely for the attention and use of the named addressee and = may be confidential. If you are not the intended recipient, you are = reminded that the information remains the property of the sender. You = must not use, disclose, distribute, copy, print or rely on this e-mail. = If you have received this message in error, please contact the sender = immediately and irrevocably delete this message and any copies. >=20 > From: aaron morton [mailto:aaron@thelastpickle.com]=20 > Sent: Wednesday, February 01, 2012 21:51 > To: user@cassandra.apache.org > Subject: Re: Consurrent compactors > =20 > (Assuming 1.0* release) > =46rom the comments in cassandra.yaml > =20 > # Number of simultaneous compactions to allow, NOT including > # validation "compactions" for anti-entropy repair. Simultaneous > # compactions can help preserve read performance in a mixed read/write > # workload, by mitigating the tendency of small sstables to accumulate > # during a single long running compactions. The default is usually > # fine and if you experience problems with compaction running too > # slowly or too fast, you should look at > # compaction_throughput_mb_per_sec first. > # > # This setting has no effect on LeveledCompactionStrategy. > # > # concurrent_compactors defaults to the number of cores. > # Uncomment to make compaction mono-threaded, the pre-0.8 default. > #concurrent_compactors: 1 > =20 > If you set it to 1 then only 1 compaction should run at a time, = excluding validation.=20 > =20 > How often do you run a cleanup compaction ? They are only necessary = when you perform a token move. > =20 > Cheers > =20 > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > =20 > On 1/02/2012, at 9:48 PM, Viktor Jevdokimov wrote: >=20 >=20 > Hi, > =20 > When concurrent compactors are set to more then 1, it=92s rare when = more than 1 compaction is running in parallel. > =20 > Didn=92t checked the source code, but it looks like when next = compaction task (any of minor, major, or cleanup) is for the same CF, it = will not start in parallel and next tasks are not checked. > =20 > Will it be possible to check all tasks, not only the next one, to find = which of them can be started? > =20 > This is actual especially when nightly cleanup is running, a lot of = cleanup tasks are pending, regular minor compactions are waiting until = all cleanup compactions are finished. > =20 > =20 > =20 > Best regards/ Pagarbiai > =20 > Viktor Jevdokimov > Senior Developer > =20 > Email: Viktor.Jevdokimov@adform.com > Phone: +370 5 212 3063. Fax: +370 5 261 0453 > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania > =20 > =20 > >=20 > > Follow: >=20 > > Visit our blog > Disclaimer: The information contained in this message and attachments = is intended solely for the attention and use of the named addressee and = may be confidential. If you are not the intended recipient, you are = reminded that the information remains the property of the sender. You = must not use, disclose, distribute, copy, print or rely on this e-mail. = If you have received this message in error, please contact the sender = immediately and irrevocably delete this message and any copies. > =20 > =20 --Apple-Mail=_48311E22-1CAC-4B8C-A583-4F613709410B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
Not sure I understand the question. Do you = have an example where a CF is not getting compacted = ? 

The compaction tasks will be processed in = the order they are submitted. If you have concurrent_compactors > 1 = then the thread pool for compactions (excluding validation compactions) = will be able to process multiple compaction tasks in = parallel. 

If you have a CF that gets a lot more = traffic than other CF's it will require more compaction. But by running = concurrent_compactors > 1 smaller CF's should still be able to get = through. 

Cheers

http://www.thelastpickle.com

On 3/02/2012, at 9:15 PM, Viktor Jevdokimov = wrote:

My = concern is not anout cleanup, but about supposed =84tendency of = small sstables to accumulate during a single long running = compactions=93. When next task is for the = same column family as currently long-running compaction, other column = families compactions are freezed and concurrent_compactors = > 1setting just not = working.
 
Best regards/ = Pagarbiai
Viktor = Jevdokimov
Senior = Developer
J. = Jasinskio 16C, LT-01112 Vilnius, = Lithuania
 

Visit our = blog


<= /table>

Disclaimer: The = information contained in this message and attachments is intended solely = for the attention and use of the named addressee and may be = confidential. If you are not the intended recipient, you are reminded = that the information remains the property of the sender. You must not = use, disclose, distribute, copy, print or rely on this e-mail. If you = have received this message in error, please contact the sender = immediately and irrevocably delete this message and any = copies.


From: aaron morton = [mailto:aaron@thelastpickle.com] 
Sent: Wednesday, February 01, = 2012 21:51
To: user@cassandra.apache.orgSubject: Re: = Consurrent compactors
 
(Assuming 1.0* = release)
=46rom the comments in = cassandra.yaml
# Number of = simultaneous compactions to allow, NOT = including
# validation "compactions" = for anti-entropy repair. =  Simultaneous
# compactions = can help preserve read performance in a mixed = read/write
# workload, by mitigating = the tendency of small sstables to = accumulate
# during a single long = running compactions. The default is = usually
# fine and if you experience = problems with compaction running too
# slowly or too fast, you should look = at
# = compaction_throughput_mb_per_sec first.
#
# This setting = has no effect on = LeveledCompactionStrategy.
#
# = concurrent_compactors defaults to the number of = cores.
# Uncomment to make = compaction mono-threaded, the pre-0.8 = default.
#concurrent_compactors: = 1
If you set it to 1 then only = 1 compaction should run at a time, excluding = validation. 
How often do you run a cleanup compaction ? They are only = necessary when you perform a token = move.
Cheers
Aaron = Morton
Freelance Developer
 

Follow:

Visit our blogDisclaimer: The information contained in = this message and attachments is intended solely for the attention and = use of the named addressee and may be confidential. If you are not the = intended recipient, you are reminded that the information remains the = property of the sender. You must not use, disclose, distribute, copy, = print or rely on this e-mail. If you have received this message in = error, please contact the sender immediately and irrevocably delete this = message and any copies.

 
 


<= /html>= --Apple-Mail=_48311E22-1CAC-4B8C-A583-4F613709410B--