Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4D057109F6 for ; Fri, 16 Jan 2015 14:15:28 +0000 (UTC) Received: (qmail 63319 invoked by uid 500); 16 Jan 2015 14:15:26 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 63280 invoked by uid 500); 16 Jan 2015 14:15:26 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 63270 invoked by uid 99); 16 Jan 2015 14:15:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jan 2015 14:15:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mightye@gmail.com designates 209.85.218.41 as permitted sender) Received: from [209.85.218.41] (HELO mail-oi0-f41.google.com) (209.85.218.41) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jan 2015 14:15:00 +0000 Received: by mail-oi0-f41.google.com with SMTP id i138so17499276oig.0 for ; Fri, 16 Jan 2015 06:14:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=P/guTvZxg63bndGk4wjqtaHBNDuRYpjqDXuHGzbyiS8=; b=EuRgRcdvUcgCf003WOTXI01M+J3XuGhPicKZmg2uKccztlVhbqdEafzd+cZa3brg8u dRF7T6kDv4j4JOIv8sHH1Gduu6b/zGpUaqDJk8n4mv73NFzMw8G8MaBxgs+zzs4Bmra8 QxWXEEh4q9D4t8sgJkLRtp11bGh8DiBtRq6IlfD9rhjgI51qsrb1AtTdnIha+tqicUuT 2l52knm2Ng2tCaiHUmDX4CzyOoVW4l/hzhb3+Zc2r60zDVW7yIsBL810uyhiNvHr0xSV D8OXF7hI0OeiRV1PjmWBNCvYdeXgWFZZ6QpUatZlpGlCKTdYdngoCJTo1IA/uvLSPoyK pXIA== X-Received: by 10.60.92.5 with SMTP id ci5mr9497533oeb.26.1421417654100; Fri, 16 Jan 2015 06:14:14 -0800 (PST) MIME-Version: 1.0 Received: by 10.76.34.71 with HTTP; Fri, 16 Jan 2015 06:13:53 -0800 (PST) In-Reply-To: <54B8BFC3.1070203@enercast.de> References: <54B78432.7010003@t-online.de> <54B8BFC3.1070203@enercast.de> From: Eric Stevens Date: Fri, 16 Jan 2015 07:13:53 -0700 Message-ID: Subject: Re: Many really small SSTables To: "user@cassandra.apache.org" Cc: j.kesten@enercast.de Content-Type: multipart/alternative; boundary=047d7b33d97459aaea050cc59668 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33d97459aaea050cc59668 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable There's another thread going on right now in this list about compactions not happening when they seemingly should. Tyler Hobbs postulates a bug and workaround for it, so maybe try that out, and if that fixes anything for you, certainly let him know. The bug Tyler postulates on is triggered when you have a write-heavy with zero read workload, and if you're testing data loading, maybe you're triggering that. Also, it's probably a long shot, but make sure that your SSTable counts haven't gone down since you last looked, if you're load testing your cluster and throwing big bursts of writes at it, you can temporarily fall behind on compaction (getting a large number of sstables), and if you stop writes, you can catch back up again as a result - maybe you looked at table counts during loading, and looked at compactionstats during a quiet time. On Fri, Jan 16, 2015 at 12:37 AM, Jan Kesten wrote: > Hi Eric and all, > > I almost expected this kind answer. I did a nodetool compactionstats > already to see if those sstables are beeing compacted, but on all nodes > there are 0 outstanding compactions (right now in the morning, not runnin= g > any tests on this cluster). > > The reported read latency is about 1-3ms and on nodes which have many > sstables (new highscore are ~18k sstables). The 99% percentile is about > 30-40 micros and a cell count of about 80-90 (if I got the docs right the= se > are the number of sstables accessed, that changed from 2.0 to 2.1 I think > as I see this only on testing cluster). > > I looks to me that compactions were not triggered. I tried a nodetool > compact on one node overnight - but that crashed the entire node. > > Roland > > Am 15.01.2015 um 19:14 schrieb Eric Stevens: > > Yes, many sstables can have a huge negative impact read performance, and > will also create memory pressure on that node. > > There are a lot of things which can produce this effect, and it strongly > also suggests you're falling behind on compaction in general (check > nodetool compactionstats, you should have <5 outstanding/pending, > preferably 0-1). To see whether and how much it is impacting your read > performance, check nodetool cfstats and nodetool > cfhistograms . > > > On Thu, Jan 15, 2015 at 2:11 AM, Roland Etzenhammer < > r.etzenhammer@t-online.de> wrote: > >> Hi, >> >> I'm testing around with cassandra fair a bit, using 2.1.2 which I know >> has some major issues,but it is a test environment. After some bulk >> loading, testing with incremental repairs and running out of heap once I >> found that now I have a quit large number of sstables which are really >> small: >> >> <1k 0 0,0% >> <10k 2780 76,8% >> <100k 3392 93,7% >> <1000k 3461 95,6% >> <10000k 3471 95,9% >> <100000k 3517 97,1% >> <1000000k 3596 99,3% >> all 3621 100,0% >> >> 76,8% of all sstables in this particular column familiy are smaller that >> 10kB, 93.7% are smaller then 100kB. >> >> Just for my understanding - does that impact performance? And is there >> any way to reduce the number of sstables? A full run of nodetool compact= is >> running for a really long time (more than 1day). >> >> Thanks for any input, >> Roland >> > > > -- > i.A. Jan Kesten Systemadministration enercast GmbH Friedrich - Ebert - > Stra=C3=9Fe 104 D=E2=80=9334119 Kassel Tel.: +49 561 / 4739664-0 Fax: > (+49)561/4739664-9 mailto: j.kesten@enercast.de http://www.enercast.de AG > Kassel HRB 15471 Thomas Landgraf Gesch=C3=A4ftsf=C3=BChrer t.landgraf@ene= rcast.de > Tel.: (+49)561/4739664-0 FAX: -9 Mobil: (+49)172/6565087 enercast GmbH > Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enercast.de > Online-Prognosen f=C3=BCr erneuerbare Energien Gesch=C3=A4ftsf=C3=BChrung= : Thomas Landgraf > (CEO), Bernd Kratz (CTO), Philipp Rinder (CSO) Diese E-Mail und etwaige > Anh=C3=A4nge k=C3=B6nnen vertrauliche und/oder rechtlich gesch=C3=BCtzte = Informationen > enthalten. Falls Sie nicht der angegebene Empf=C3=A4nger sind oder falls = diese > E-Mail irrt=C3=BCmlich an Sie adressiert wurde, benachrichtigen Sie uns b= itte > sofort durch Antwort-E-Mail und l=C3=B6schen Sie diese E-Mail nebst etwai= gen > Anlagen von Ihrem System. Ebenso d=C3=BCrfen Sie diese E-Mail oder ihre A= nlagen > nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and a= ny > attachment may contain confidential and/or privileged information. If you > are not the named addressee or if this transmission has been addressed to > you in error, please notify us immediately by reply e-mail and then delet= e > this e-mail and any attachment from your system. Please understand that y= ou > must not copy this e-mail or any attachment or disclose the contents to a= ny > other person. Thank you for your cooperation. > --047d7b33d97459aaea050cc59668 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
There's another thread going on right now in this list= about compactions not happening when they seemingly should.=C2=A0 Tyler Ho= bbs postulates a bug and workaround for it, so maybe try that out, and if t= hat fixes anything for you, certainly let him know.=C2=A0 The bug Tyler pos= tulates on is triggered when you have a write-heavy with zero read workload= , and if you're testing data loading, maybe you're triggering that.=

Also, it's probably a long shot, but make sure that= your SSTable counts haven't gone down since you last looked, if you= 9;re load testing your cluster and throwing big bursts of writes at it, you= can temporarily fall behind on compaction (getting a large number of sstab= les), and if you stop writes, you can catch back up again as a result - may= be you looked at table counts during loading, and looked at compactionstats= during a quiet time.

On Fri, Jan 16, 2015 at 12:37 AM, Jan Kesten <j.kesten= @enercast.de> wrote:
=20 =20 =20
Hi Eric and all,

I almost expected this kind answer. I did a nodetool compactionstats already to see if those sstables are beeing compacted, but on all nodes there are 0 outstanding compactions (right now in the morning, not running any tests on this cluster).

The reported read latency is about 1-3ms and on nodes which have many sstables (new highscore are ~18k sstables). The 99% percentile is about 30-40 micros and a cell count of about 80-90 (if I got the docs right these are the number of sstables accessed, that changed from 2.0 to 2.1 I think as I see this only on testing cluster).

I looks to me that compactions were not triggered. I tried a nodetool compact on one node overnight - but that crashed the entire node.

Roland

Am 15.01.2015 um 19:14 schrieb Eric Stevens:
Yes, many sstables can have a huge negative impact read performance, and will also create memory pressure on that node.

There are a lot of things which can produce this effect, and it strongly also suggests you're falling behind on compaction in general (check nodetool compactionstats, you should have <5 outstanding/pending, preferably 0-1).=C2=A0 To see whether and how much it is impacting your read performance, check nodetool cfstats <keyspace.table> and nodetool cfhistograms <keyspace> <table>.


On Thu, Jan 15, 2015 at 2:11 AM, Roland Etzenhammer <r.etzenhammer@t-online.de> wrote:
Hi,

I'm testing around with cassandra fair a bit, using 2.1.2 which I know has some major issues,but it is a test environment. After some bulk loading, testing with incremental repairs and running out of heap once I found that now I have a quit large number of sstables which are really small:

<1k=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 = =C2=A0 =C2=A0 0,0%
<10k=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2780=C2=A0 =C2=A0 =C2= =A076,8%
<100k=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03392=C2=A0 =C2=A0 =C2= =A093,7%
<1000k=C2=A0 =C2=A0 =C2=A0 =C2=A0 3461=C2=A0 =C2=A0 =C2=A095= ,6%
<10000k=C2=A0 =C2=A0 =C2=A0 =C2=A03471=C2=A0 =C2=A0 =C2=A095= ,9%
<100000k=C2=A0 =C2=A0 =C2=A0 3517=C2=A0 =C2=A0 =C2=A097,1% <1000000k=C2=A0 =C2=A0 =C2=A03596=C2=A0 =C2=A0 =C2=A099,3% all=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03621=C2=A0 =C2=A0 1= 00,0%

76,8% of all sstables in this particular column familiy are smaller that 10kB, 93.7% are smaller then 100kB.

Just for my understanding - does that impact performance? And is there any way to reduce the number of sstables? A full run of nodetool compact is running for a really long time (more than 1day).

Thanks for any input,
Roland


--
i.A. Jan Kesten Systemadministration enercast GmbH Friedrich - Ebert - Stra=C3=9Fe 104 D=E2=80=9334119 Kassel Tel.: +49 561 / 4739664-0 Fax: (+49)561/4739664-9 mailto: j.k= esten@enercast.de http://www.enerc= ast.de AG Kassel HRB 15471 Thomas Landgraf Gesch=C3=A4ftsf=C3=BChrer t.landgra= f@enercast.de Tel.: (+49)561/4739664-0 FAX: -9 Mobil: (+49)1= 72/6565087 enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enerc= ast.de Online-Prognosen f=C3=BCr erneuerbare Energien Gesch=C3=A4ftsf=C3=BChrung: Thomas Landgraf (CEO), Bernd Kratz (CTO), Philipp Rinder (CSO) Diese E-Mail und etwaige Anh=C3=A4nge k=C3=B6nnen vertrauliche und/od= er rechtlich gesch=C3=BCtzte Informationen enthalten. Falls Sie nicht de= r angegebene Empf=C3=A4nger sind oder falls diese E-Mail irrt=C3=BCmlic= h an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und l=C3=B6schen Sie diese E-Mail nebst etwaigen Anlag= en von Ihrem System. Ebenso d=C3=BCrfen Sie diese E-Mail oder ihre Anlag= en nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.

--047d7b33d97459aaea050cc59668--