cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Stevens <migh...@gmail.com>
Subject Re: Many really small SSTables
Date Fri, 16 Jan 2015 14:13:53 GMT
There's another thread going on right now in this list about compactions
not happening when they seemingly should.  Tyler Hobbs postulates a bug and
workaround for it, so maybe try that out, and if that fixes anything for
you, certainly let him know.  The bug Tyler postulates on is triggered when
you have a write-heavy with zero read workload, and if you're testing data
loading, maybe you're triggering that.

Also, it's probably a long shot, but make sure that your SSTable counts
haven't gone down since you last looked, if you're load testing your
cluster and throwing big bursts of writes at it, you can temporarily fall
behind on compaction (getting a large number of sstables), and if you stop
writes, you can catch back up again as a result - maybe you looked at table
counts during loading, and looked at compactionstats during a quiet time.

On Fri, Jan 16, 2015 at 12:37 AM, Jan Kesten <j.kesten@enercast.de> wrote:

>  Hi Eric and all,
>
> I almost expected this kind answer. I did a nodetool compactionstats
> already to see if those sstables are beeing compacted, but on all nodes
> there are 0 outstanding compactions (right now in the morning, not running
> any tests on this cluster).
>
> The reported read latency is about 1-3ms and on nodes which have many
> sstables (new highscore are ~18k sstables). The 99% percentile is about
> 30-40 micros and a cell count of about 80-90 (if I got the docs right these
> are the number of sstables accessed, that changed from 2.0 to 2.1 I think
> as I see this only on testing cluster).
>
> I looks to me that compactions were not triggered. I tried a nodetool
> compact on one node overnight - but that crashed the entire node.
>
> Roland
>
> Am 15.01.2015 um 19:14 schrieb Eric Stevens:
>
> Yes, many sstables can have a huge negative impact read performance, and
> will also create memory pressure on that node.
>
>  There are a lot of things which can produce this effect, and it strongly
> also suggests you're falling behind on compaction in general (check
> nodetool compactionstats, you should have <5 outstanding/pending,
> preferably 0-1).  To see whether and how much it is impacting your read
> performance, check nodetool cfstats <keyspace.table> and nodetool
> cfhistograms <keyspace> <table>.
>
>
> On Thu, Jan 15, 2015 at 2:11 AM, Roland Etzenhammer <
> r.etzenhammer@t-online.de> wrote:
>
>> Hi,
>>
>> I'm testing around with cassandra fair a bit, using 2.1.2 which I know
>> has some major issues,but it is a test environment. After some bulk
>> loading, testing with incremental repairs and running out of heap once I
>> found that now I have a quit large number of sstables which are really
>> small:
>>
>> <1k              0      0,0%
>> <10k          2780     76,8%
>> <100k         3392     93,7%
>> <1000k        3461     95,6%
>> <10000k       3471     95,9%
>> <100000k      3517     97,1%
>> <1000000k     3596     99,3%
>> all           3621    100,0%
>>
>> 76,8% of all sstables in this particular column familiy are smaller that
>> 10kB, 93.7% are smaller then 100kB.
>>
>> Just for my understanding - does that impact performance? And is there
>> any way to reduce the number of sstables? A full run of nodetool compact is
>> running for a really long time (more than 1day).
>>
>> Thanks for any input,
>> Roland
>>
>
>
> --
> i.A. Jan Kesten Systemadministration enercast GmbH Friedrich - Ebert -
> Straße 104 D–34119 Kassel Tel.: +49 561 / 4739664-0 Fax:
> (+49)561/4739664-9 mailto: j.kesten@enercast.de http://www.enercast.de AG
> Kassel HRB 15471 Thomas Landgraf Geschäftsführer t.landgraf@enercast.de
> Tel.: (+49)561/4739664-0 FAX: -9 Mobil: (+49)172/6565087 enercast GmbH
> Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enercast.de
> Online-Prognosen für erneuerbare Energien Geschäftsführung: Thomas Landgraf
> (CEO), Bernd Kratz (CTO), Philipp Rinder (CSO) Diese E-Mail und etwaige
> Anhänge können vertrauliche und/oder rechtlich geschützte Informationen
> enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese
> E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte
> sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen
> Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen
> nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any
> attachment may contain confidential and/or privileged information. If you
> are not the named addressee or if this transmission has been addressed to
> you in error, please notify us immediately by reply e-mail and then delete
> this e-mail and any attachment from your system. Please understand that you
> must not copy this e-mail or any attachment or disclose the contents to any
> other person. Thank you for your cooperation.
>

Mime
View raw message