cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Subject Re: failing bootstraps with OOM
Date Thu, 03 Nov 2016 14:46:18 GMT
On Thu, Nov 3, 2016 at 2:32 PM, Mike Torra <mtorra@demandware.com> wrote:

> Hi Alex - I do monitor sstable counts and pending compactions, but
> probably not closely enough. In 3/4 regions the cluster is running in, both
> counts are very high - ~30-40k sstables for one particular CF, and on many
> nodes >1k pending compactions.
>

It is generally a good idea to try to keep the number of pending
compactions minimal.  We usually see it is close to zero on every node
during normal operations and less than some tens during maintenance such as
repair.

I had noticed this before, but I didn't have a good sense of what a "high"
> number for these values was.
>

I would say anything higher than 20 probably requires someone to have a
look and over 1k is very troublesome.

It makes sense to me why this would cause the issues I've seen. After
> increasing concurrent_compactors and compaction_throughput_mb_per_sec (to
> 8 and 64mb, respectively), I'm starting to see those counts go down
> steadily. Hopefully that will resolve the OOM issues, but it looks like it
> will take a while for compactions to catch up.
>
> Thanks for the suggestions, Alex
>

Welcome. :-)

--
Alex

Mime
View raw message