Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 84D4A106C6 for ; Wed, 24 Apr 2013 22:39:37 +0000 (UTC) Received: (qmail 57582 invoked by uid 500); 24 Apr 2013 22:39:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 57503 invoked by uid 500); 24 Apr 2013 22:39:34 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 57494 invoked by uid 99); 24 Apr 2013 22:39:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Apr 2013 22:39:34 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLYTO_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [72.30.239.74] (HELO nm34-vm2.bullet.mail.bf1.yahoo.com) (72.30.239.74) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Apr 2013 22:39:28 +0000 Received: from [98.139.212.151] by nm34.bullet.mail.bf1.yahoo.com with NNFMP; 24 Apr 2013 22:39:07 -0000 Received: from [98.139.212.233] by tm8.bullet.mail.bf1.yahoo.com with NNFMP; 24 Apr 2013 22:39:07 -0000 Received: from [127.0.0.1] by omp1042.mail.bf1.yahoo.com with NNFMP; 24 Apr 2013 22:39:07 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 706409.85696.bm@omp1042.mail.bf1.yahoo.com Received: (qmail 36933 invoked by uid 60001); 24 Apr 2013 22:39:07 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1366843147; bh=ahnECoW+PWjXTQEXarkq5zJFyNb0BrbHeEYYUThblXA=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=YpkEqqz/FFUbIcxESgceNTvzAK6eef2N+kdVcHRk7caXuvY5BFdcp6TTUPg8kZrvFkh98GcqN0Ur5QLLTmNj4cP1yT37eNX2QKqUPOzx6Y/6fNfFUEHozGvR/84KbZ0EWp6WUsfZxjv34NldxkpwaCyV2PPko6jFaf7LPf4tNZw= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=sIcvVUnCGqgfwTQstP6CkkEtAX8MxuKveLUz4S2I5lcCRLGarMPQYKzeetXdOT+fAongcswYKUbBMmJ3yDszeet8t8D8K7a3IcEwqQSK94m9agf9Jhg+zaFyrIQsxNl16hr279oexyLXU/wcezToY/7jZ/XdTTlD+hJPDWbd0sc=; X-YMail-OSG: rRusN.kVM1lTyI7J8mQQaWs5AzOrbJXQkuI5tKdrMiKVyyb LN9fj8AF1i7v7hgQ2sSGkArS3p.Ofd1Hy74SObIE.AsEBlH08HRNzel1.xhZ hC0hRQ1IlpfjrCH.4QACATxfduB1v6v7papXiONgYXYz.QYWp89Fups1zgKj HSYKLrWx9OF7ufSsJWZmMKxuq2Xp9E7iNnK_zfHIZMwRr7ZzNxJBj8e.mBZ3 CCmISEggbBL12BbqGaj2bboOEBz9crMwbKe7M.NRwy25z6tDOunljNPhueXM k7ik34o9JiEo_CRMbnqqhkVrybrB2OG3BTNFboQajVK1WdaLmTE1q4P.Sczx bAYH_zoSyBguF6hYV2uf6KG5WZ406bwWKmaSIIR7D8Pdo1ZpVa_HKS3myJ0Z Mewt8X.aBBcWeMD6m43JlbcjJ5zZqp8ijwcM8VuX86wqM2RpL3zoTcpg4FZo X59WewduKvpPZOm9zCfhSO3O1QuCVvkeXd5odXQUCCWcQzw9zxehYQnkUE6g ovGWOxxs5_BeY Received: from [208.185.20.30] by web160905.mail.bf1.yahoo.com via HTTP; Wed, 24 Apr 2013 15:39:07 PDT X-Rocket-MIMEInfo: 002.001,U2FtZSBoZXJlLiBXZSBkaXNhYmxlIHRoZSB0aHJvdHRsaW5nIGFuZCBvdXIgZGlzayBhbmQgQ1BVIHVzYWdlIGJvdGggbG93ICg8IDEwJSkgYW5kIHN0aWxsIHRha2VzIGhvdXJzIGZvciBMQ1MgY29tcGFjdGlvbiB0byBmaW5pc2ggYWZ0ZXIgYSByZXBhaXIuIEZvciB0aGlzIGNsdXN0ZXIsIHdlIGRvbid0IGRlbGV0ZSBhbnkgZGF0YSwgc28gd2UgY2FuIHJ1bGUgb3V0IHRvbWJzdG9uZXMuIE5vdCBzdXJlIHdoYXQgaXMgaG9sZGluZyBjb21wYWN0aW9uIGJhY2suIE15IG9ic2VydmF0aW9uIGlzIHRoYXQgZm8BMAEBAQE- X-Mailer: YahooMailWebService/0.8.141.536 References: Message-ID: <1366843147.36745.YahooMailNeo@web160905.mail.bf1.yahoo.com> Date: Wed, 24 Apr 2013 15:39:07 -0700 (PDT) From: Wei Zhu Reply-To: Wei Zhu Subject: Re: compaction throughput rate not even close to 16MB To: "user@cassandra.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-1879531-1645910741-1366843147=:36745" X-Virus-Checked: Checked by ClamAV on apache.org ---1879531-1645910741-1366843147=:36745 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Same here. We disable the throttling and our disk and CPU usage both low (<= 10%) and still takes hours for LCS compaction to finish after a repair. Fo= r this cluster, we don't delete any data, so we can rule out tombstones. No= t sure what is holding compaction back. My observation is that for the LCS = which involves large number of SSTables (since we set SSTable size too smal= l at 10M and sometimes one compactions involves up to 10 G of data =3D 1000= SSTables), the throughout put is smaller. So my theory is that open/close = file handlers have substantial impact on the throughput.=A0=0A=0ABy the way= , we are on SSD.=0A=0A-Wei=0A=0A________________________________=0A From: "= Hiller, Dean" =0ATo: "user@cassandra.apache.org" =0ASent: Wednesday, April 24, 2013 1:37 PM=0ASubjec= t: Re: compaction throughput rate not even close to 16MB=0A =0A=0AThanks mu= ch!!!=A0 Better to hear at least one other person sees the same thing ;).= =A0 Sometimes these posts just go silent.=0A=0ADean=0A=0AFrom: Edward Capri= olo >=0AReply-To: "user= @cassandra.apache.org" >=0ADate: Wednesday, April 24, 20= 13 2:33 PM=0ATo: "user@cassandra.apache.org" >=0ASubject= : Re: compaction throughput rate not even close to 16MB=0A=0AI have noticed= the same. I think in the "real" world your compaction throughput is limite= d by other things. If I had to speculate I would say that compaction can re= move expired tombstones, however doing this requires bloom filter checks, e= tc.=0A=0AI think that setting is more important with multi threaded compact= ion and/or more compaction slots. In those cases it may actually throttle s= omething.=0A=0A=0AOn Wed, Apr 24, 2013 at 3:54 PM, Hiller, Dean > wrote:=0AI was wondering about the= compactionthroughput.=A0 I never see ours get even close to 16MB and I tho= ught this is supposed to throttle compaction, right?=A0 Ours is constantly = less than 3MB/sec from looking at our logs or do I have this totally wrong?= =A0 How can I see the real throughput so that I can understand how to throt= tle it when I need to?=0A=0A94,940,780 bytes to 95,346,024 (~100% of origin= al) in 38,438ms =3D 2.365603MB/s.=A0 2,350,114 total rows, 2,350,022 unique= .=A0 Row merge counts were {1:2349930, 2:92, }=0A=0AThanks,=0ADean ---1879531-1645910741-1366843147=:36745 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Same here. We disable= the throttling and our disk and CPU usage both low (< 10%) and still ta= kes hours for LCS compaction to finish after a repair. For this cluster, we= don't delete any data, so we can rule out tombstones. Not sure what is hol= ding compaction back. My observation is that for the LCS which involves lar= ge number of SSTables (since we set SSTable size too small at 10M and somet= imes one compactions involves up to 10 G of data =3D 1000 SSTables), the th= roughout put is smaller. So my theory is that open/close file handlers have= substantial impact on the throughput. 

By th= e way, we are on SSD.

-Wei
From: "Hiller, Dean" <Dean.Hiller@nrel= .gov>
To: "user@cas= sandra.apache.org" <user@cassandra.apache.org>
Sent: Wednesday, April 24, 2013 1:37 PM
= Subject: Re: compaction t= hroughput rate not even close to 16MB

Thanks much!!!  Better to hear at least= one other person sees the same thing ;).  Sometimes these posts just = go silent.

Dean

From: Edward Capriolo <edlinuxguru@= gmail.com<mailto:edlinuxguru@gmail.com>>
Reply-T= o: "user@cassandra.apache.org<mailto:u= ser@cassandra.apache.org>" <user@cassandra.apache.= org<mailto:user@cassandra.apache.org>>
D= ate: Wednesday, April 24, 2013 2:33 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra= .apache.org>>
Subject: Re: compaction throughput rate not even= close to 16MB

I have noticed the same. I think in the "real" world = your compaction throughput is limited by other things. If I had to speculat= e I would say that compaction can remove expired tombstones, however doing = this requires bloom filter checks, etc.

I think that setting is more= important with multi threaded compaction and/or more compaction slots. In = those cases it may actually throttle something.


On Wed, Apr 24, 2013 = at 3:54 PM, Hiller, Dean <Dean.Hiller@nrel.gov<mailto:D= ean.Hiller@nrel.gov>> wrote:
I was wondering about the compact= ionthroughput.  I never see ours get even close to 16MB and I thought = this is supposed to throttle compaction, right?  Ours is constantly le= ss than 3MB/sec from looking at our logs or do I have this totally wrong?&n= bsp; How can I see the real throughput so that I can understand how to thro= ttle it when I need to?

94,940,780 bytes to 95,346,024 (~100% of ori= ginal) in 38,438ms =3D 2.365603MB/s.  2,350,114 total rows, 2,350,022 = unique.  Row merge counts were {1:2349930, 2:92, }

Thanks,
D= ean





---1879531-1645910741-1366843147=:36745--