cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Kesten <>
Subject Re: Many really small SSTables
Date Fri, 16 Jan 2015 07:37:39 GMT
Hi Eric and all,

I almost expected this kind answer. I did a nodetool compactionstats 
already to see if those sstables are beeing compacted, but on all nodes 
there are 0 outstanding compactions (right now in the morning, not 
running any tests on this cluster).

The reported read latency is about 1-3ms and on nodes which have many 
sstables (new highscore are ~18k sstables). The 99% percentile is about 
30-40 micros and a cell count of about 80-90 (if I got the docs right 
these are the number of sstables accessed, that changed from 2.0 to 2.1 
I think as I see this only on testing cluster).

I looks to me that compactions were not triggered. I tried a nodetool 
compact on one node overnight - but that crashed the entire node.


Am 15.01.2015 um 19:14 schrieb Eric Stevens:
> Yes, many sstables can have a huge negative impact read performance, 
> and will also create memory pressure on that node.
> There are a lot of things which can produce this effect, and it 
> strongly also suggests you're falling behind on compaction in general 
> (check nodetool compactionstats, you should have <5 
> outstanding/pending, preferably 0-1).  To see whether and how much it 
> is impacting your read performance, check nodetool cfstats 
> <keyspace.table> and nodetool cfhistograms <keyspace> <table>.
> On Thu, Jan 15, 2015 at 2:11 AM, Roland Etzenhammer 
> < <>> wrote:
>     Hi,
>     I'm testing around with cassandra fair a bit, using 2.1.2 which I
>     know has some major issues,but it is a test environment. After
>     some bulk loading, testing with incremental repairs and running
>     out of heap once I found that now I have a quit large number of
>     sstables which are really small:
>     <1k              0      0,0%
>     <10k          2780     76,8%
>     <100k         3392     93,7%
>     <1000k        3461     95,6%
>     <10000k       3471     95,9%
>     <100000k      3517     97,1%
>     <1000000k     3596     99,3%
>     all           3621    100,0%
>     76,8% of all sstables in this particular column familiy are
>     smaller that 10kB, 93.7% are smaller then 100kB.
>     Just for my understanding - does that impact performance? And is
>     there any way to reduce the number of sstables? A full run of
>     nodetool compact is running for a really long time (more than 1day).
>     Thanks for any input,
>     Roland

i.A. Jan Kesten Systemadministration enercast GmbH Friedrich - Ebert - 
Straße 104 D–34119 Kassel Tel.: +49 561 / 4739664-0 Fax: 
(+49)561/4739664-9 mailto: 
AG Kassel HRB 15471 Thomas Landgraf Geschäftsführer Tel.: (+49)561/4739664-0 FAX: -9 Mobil: 
(+49)172/6565087 enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel 
HRB15471 Online-Prognosen für erneuerbare 
Energien Geschäftsführung: Thomas Landgraf (CEO), Bernd Kratz (CTO), 
Philipp Rinder (CSO) Diese E-Mail und etwaige Anhänge können 
vertrauliche und/oder rechtlich geschützte Informationen enthalten. 
Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail 
irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort 
durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen 
von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht 
kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any 
attachment may contain confidential and/or privileged information. If 
you are not the named addressee or if this transmission has been 
addressed to you in error, please notify us immediately by reply e-mail 
and then delete this e-mail and any attachment from your system. Please 
understand that you must not copy this e-mail or any attachment or 
disclose the contents to any other person. Thank you for your cooperation.

View raw message