Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B6B0B186B9 for ; Thu, 21 Jan 2016 15:12:34 +0000 (UTC) Received: (qmail 23130 invoked by uid 500); 21 Jan 2016 15:12:32 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23094 invoked by uid 500); 21 Jan 2016 15:12:32 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23084 invoked by uid 99); 21 Jan 2016 15:12:32 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2016 15:12:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C838EC13D0 for ; Thu, 21 Jan 2016 15:12:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.121 X-Spam-Level: *** X-Spam-Status: No, score=3.121 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_HUGEIMGSRC=0.2, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001, URI_TRY_3LD=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=datastax.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id EttRl1hO28vf for ; Thu, 21 Jan 2016 15:12:18 +0000 (UTC) Received: from mail-wm0-f46.google.com (mail-wm0-f46.google.com [74.125.82.46]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 4E371256E0 for ; Thu, 21 Jan 2016 15:12:17 +0000 (UTC) Received: by mail-wm0-f46.google.com with SMTP id r129so176218964wmr.0 for ; Thu, 21 Jan 2016 07:12:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datastax.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=FGMs5hObTl8NJd11Puz2yVzEWYDDyvJYIYYQ8nTBdhw=; b=YVGxk2s2naYYGSlGVWLYeq/0IoWtDLgpN0wY7uIIuq4dYHnNCm8cNwc1+zWbveTzIp b352j3WT6M5ayvI0xFpEGlMnaEiTAB8nt/j9gmAnDyDrTzzwuBbNXIh0sR9t5lSjl806 rnZCopYURwoDtsSNeIVafojmypwcpIrVCw5Qo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=FGMs5hObTl8NJd11Puz2yVzEWYDDyvJYIYYQ8nTBdhw=; b=W7U+RI84Ga4saFvAOZBerl0070kCRS6zoWAy3nZp4Xq2xxh78kWZtZmjKYOIxsNrNy dM2HQPHHCNfwlaKvT2Wlkf3HQhjQwx7xoUI2dV3zciVBhSyImTj7qdNIFixxbkEry2BO PQnow+Yrv+2Glf3/R9JZdWdASenxTrtHsptyaxMBJjrzEDK2sK9Q9C5bN71xfIKLTypd pTl13V9Ptfxh4eEQvSGzgRCxlYBS2Ke0CuS+FsfXvWRPFx8xkD2D58yBn3qrJ2OVMex5 metkSvBv5BZ+nM+rPDpZQHFzy2JJTurwQ3pz6SQSnCmUcAXOH9UxiKAMKcdPUL2dihhv WdKA== X-Gm-Message-State: ALoCoQki87zBTbWe/u4AucRI39ddnJn44V+6kR9llozBLkEPa97Lsf+mDsugJ8o7AvuEjDKVlDGR5yhAa1eazOwTS4ZlT0O7xaT0KULGKWF3a+33uhEw120= X-Received: by 10.194.123.167 with SMTP id mb7mr50392444wjb.0.1453389129298; Thu, 21 Jan 2016 07:12:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.170.169 with HTTP; Thu, 21 Jan 2016 07:11:49 -0800 (PST) In-Reply-To: References: <47C33AA2-0DA3-4364-9A2C-DCF07951C67C@crowdstrike.com> From: Sebastian Estevez Date: Thu, 21 Jan 2016 10:11:49 -0500 Message-ID: Subject: Re: compaction throughput To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0115fc28c5d35c0529d986d4 --089e0115fc28c5d35c0529d986d4 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable @penguin There have been steady improvements in the different compaction strategies recently but not major re-writes. All the best, [image: datastax_logo.png] Sebasti=C3=A1n Est=C3=A9vez Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com [image: linkedin.png] [image: facebook.png] [image: twitter.png] [image: g+.png] DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world=E2=80=99s most innovative enterpri= ses. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. On Thu, Jan 21, 2016 at 9:12 AM, Kai Wang wrote: > I am using 2.2.4 and have seen multiple compactors running on the same > table. The number of compactors seems to be controlled by > concurrent_compactors. As of type of compactions, I've seen normal > compaction, tombstone compaction. Validation and Anticompaction seem to > always be single threaded. > > On Thu, Jan 21, 2016 at 8:28 AM, PenguinWhispererThe . < > th3penguinwhisperer@gmail.com> wrote: > >> Thanks for that clarification Sebastian! That's really good to know! I >> never took increasing this value in consideration because of my previous >> experience. >> >> In my case I had a table that was compacting over and over... and only >> one CPU was used. So that made me believe it was not multithreaded (I >> actually believe I asked this on IRC however it's been a few months ago = so >> I might be wrong). >> >> Have there been behavioral changes on this lately? (I was using 2.0.9 or >> 2.0.11 I believe). >> >> 2016-01-21 14:15 GMT+01:00 Sebastian Estevez < >> sebastian.estevez@datastax.com>: >> >>> >So compaction of one table will NOT spread over different cores. >>> >>> This is not exactly true. You actually can have multiple compactions >>> running at the same time on the same table, it just doesn't happen all = that >>> often. You essentially would have to have two sets of sstables that are >>> both eligible for compactions at the same time. >>> >>> all the best, >>> >>> Sebasti=C3=A1n >>> On Jan 21, 2016 7:41 AM, "PenguinWhispererThe ." < >>> th3penguinwhisperer@gmail.com> wrote: >>> >>>> After having some issues myself with compaction I think it's noteworth= y >>>> to explicitly state that compaction of a table can only run on one CPU= . So >>>> compaction of one table will NOT spread over different cores. >>>> To really have use of concurrent_compactors you need to have multiple >>>> table compactions initiated at the same time. If those are small they'= ll >>>> finish way earlier resulting in only one core using 100% as compaction= is >>>> generally CPU bound (unless your disks can't keep up). >>>> I believe it's better to be CPU(core) bound on one core(or at least no= t >>>> all) for compaction than disk IO bound as this would result in writes = and >>>> reads, ... having performance impact. >>>> Compaction is a maintenance task so it shouldn't be eating all your >>>> resources. >>>> >>>> >>>> This >>>> email has been sent from a virus-free computer protected by Avast. >>>> www.avast.com >>>> >>>> <#-1919795192_-2069969251_1162782367_-1582318301_DDB4FAA8-2DD7-40BB-A1= B8-4E2AA1F9FDF2> >>>> >>>> 2016-01-16 0:18 GMT+01:00 Kai Wang : >>>> >>>>> Jeff & Sebastian, >>>>> >>>>> Thanks for the reply. There are 12 cores but in my case C* only uses >>>>> one core most of the time. *nodetool compactionstats* shows there's >>>>> only one compactor running. I can see C* process only uses one core. = So I >>>>> guess I should've asked the question more clearly: >>>>> >>>>> 1. Is ~25 M/s a reasonable compaction throughput for one core? >>>>> 2. Is there any configuration that affects single core compaction >>>>> throughput? >>>>> 3. Is concurrent_compactors the only option to parallelize compaction= ? >>>>> If so, I guess it's the compaction strategy itself that decides when = to >>>>> parallelize and when to block on one core. Then there's not much we c= an do >>>>> here. >>>>> >>>>> Thanks. >>>>> >>>>> On Fri, Jan 15, 2016 at 5:23 PM, Jeff Jirsa < >>>>> jeff.jirsa@crowdstrike.com> wrote: >>>>> >>>>>> With SSDs, the typical recommendation is up to 0.8-1 compactor per >>>>>> core (depending on other load). How many CPU cores do you have? >>>>>> >>>>>> >>>>>> From: Kai Wang >>>>>> Reply-To: "user@cassandra.apache.org" >>>>>> Date: Friday, January 15, 2016 at 12:53 PM >>>>>> To: "user@cassandra.apache.org" >>>>>> Subject: compaction throughput >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to figure out the bottleneck of compaction on my node. >>>>>> The node is CentOS 7 and has SSDs installed. The table is configured= to use >>>>>> LCS. Here is my compaction related configs in cassandra.yaml: >>>>>> >>>>>> compaction_throughput_mb_per_sec: 160 >>>>>> concurrent_compactors: 4 >>>>>> >>>>>> I insert about 10G of data and start observing compaction. >>>>>> >>>>>> *nodetool compaction* shows most of time there is one compaction. >>>>>> Sometimes there are 3-4 (I suppose this is controlled by >>>>>> concurrent_compactors). During the compaction, I see one CPU core is= 100%. >>>>>> At that point, disk IO is about 20-25 M/s write which is much lower = than >>>>>> the disk is capable of. Even when there are 4 compactions running, I= see >>>>>> CPU go to +400% but disk IO is still at 20-25M/s write. I use *nodet= ool >>>>>> setcompactionthroughput 0* to disable the compaction throttle but >>>>>> don't see any difference. >>>>>> >>>>>> Does this mean compaction is CPU bound? If so 20M/s is kinda low. Is >>>>>> there anyway to improve the throughput? >>>>>> >>>>>> Thanks. >>>>>> >>>>> >>>>> >>>> >> > --089e0115fc28c5d35c0529d986d4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
@penguin There have been steady improvements in the differ= ent compaction strategies recently but not major re-writes.


On Thu, Jan 21, 2016 at 9:12 AM, Kai Wang <d= epend@gmail.com> wrote:
I am using 2.2.4 and have seen multiple compactors running on= the same table. The number of compactors seems to be controlled by concurr= ent_compactors. As of type of compactions, I've seen normal compaction,= tombstone compaction. Validation and Anticompaction seem to always be sing= le threaded.

On Thu, Jan 21, 2016 at 8:28 A= M, PenguinWhispererThe . <th3penguinwhisperer@gmail.com>= ; wrote:
Thanks for that clarification Sebastian! That's really good to know! = I never took increasing this value in consideration because of my previous = experience.

In my case I had a table that was compacting over = and over... and only one CPU was used. So that made me believe it was not m= ultithreaded (I actually believe I asked this on IRC however it's been = a few months ago so I might be wrong).

Have there been behavio= ral changes on this lately? (I was using 2.0.9 or 2.0.11 I believe).

2016= -01-21 14:15 GMT+01:00 Sebastian Estevez <sebastian.estevez@d= atastax.com>:

>So compaction of one table will NOT spread over different core= s.

This is not exactly true. You actually can have multi= ple compactions running at the same time on the same table, it just doesn&#= 39;t happen all that often. You essentially would have to have two sets of = sstables that are both eligible for compactions at the same time.

all the best,

Sebasti=C3=A1n

On Jan 21, 2016 7:41 AM, "PenguinWhispererT= he ." <th3penguinwhisperer@gmail.com> wrote:
After having some iss= ues myself with compaction I think it's noteworthy to explicitly state = that compaction of a table can only run on one CPU. So compaction of one ta= ble will NOT spread over different cores.
To really have use = of concurrent_compactors you need to have multiple table compactions initia= ted at the same time. If those are small they'll finish way earlier res= ulting in only one core using 100% as compaction is generally CPU bound (un= less your disks can't keep up).
I believe it's better= to be CPU(core) bound on one core(or at least not all) for compaction than= disk IO bound as this would result in writes and reads, ... having perform= ance impact.
Compaction is a maintenance task so it shouldn&#= 39;t be eating all your resources.

This email has been = sent from a virus-free computer protected by Avast.
www.avast.com

2016-01-16 0:18 GMT= +01:00 Kai Wang <depend@gmail.com>:
Jeff & Sebastian,
Thanks for the reply. There are 12 cores but in my case C* = only uses one core most of the time. nodetool compactionstats shows = there's only one compactor running. I can see C* process only uses one = core. So I guess I should've asked the question more clearly:
=
1. Is ~25 M/s a reasonable compaction throughput for one core?
2. Is there any configuration that affects single core compaction th= roughput?
3. Is concurrent_compactors the only option to paralleli= ze compaction? If so, I guess it's the compaction strategy itself that = decides when to parallelize and when to block on one core. Then there's= not much we can do here.

Thanks.

On Fri, Jan 15, 2016 at 5:= 23 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrot= e:
With SSDs, the typical recom= mendation is up to 0.8-1 compactor per core (depending on other load). =C2= =A0How many CPU cores= do you have?

=

From: Kai Wang
Reply-To: "user@cassandra.apache.org"
Date: Friday, January 15, 2016 at 12:53 PM
To: "user@cassandra.apache.org"
Subject: compaction throughput

Hi,<= br>
I am trying to figure out the bottleneck of compaction on my node. The node= is CentOS 7 and has SSDs installed. The table is configured to use LCS. He= re is my compaction related configs in cassandra.yaml:

compaction_throughput_mb_per_sec: 160
concurrent_compactors: 4

I insert about 10G of data and start observing compaction.

= nodetool compaction shows most of time there is one compaction. Sometim= es there are 3-4 (I suppose this is controlled by concurrent_compactors). D= uring the compaction, I see one CPU core is 100%. At that point, disk IO is= about 20-25 M/s write which is much lower than the disk is capable of. Even when there are 4 compactio= ns running, I see CPU go to +400% but disk IO is still at 20-25M/s write. I= use nodetool setcompactionthroughput 0 to disable the compaction throttl= e but don't see any difference.

Does this mean compac= tion is CPU bound? If so 20M/s is kinda low. Is there anyway to improve the= throughput?

Thanks.





--089e0115fc28c5d35c0529d986d4--