cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juho Mäkinen <>
Subject Re: My cluster shows high system load without any apparent reason
Date Fri, 22 Jul 2016 12:10:07 GMT
After a few days I've also tried disabling Linux kernel huge pages
defragement (echo never > /sys/kernel/mm/transparent_hugepage/defrag) and
turning coalescing off (otc_coalescing_strategy: DISABLED), but either did
do any good. I'm using LCS, there are no big GC pauses, and I have set
"concurrent_compactors: 5" (machines have 16 CPUs), but there are usually
not any compactions running when the load spike comes. "nodetool tpstats"
shows no running thread pools except on the Native-Transport-Requests
(usually 0-4) and perhaps ReadStage (usually 0-1).

The symptoms are the same: after about 12-24 hours increasingly number of
nodes start to show short CPU load spikes and this affects the median read
latencies. I ran a dstat when a load spike was already under way (see
screenshot, but any other column than the
load itself doesn't show any major change except the system/kernel CPU

All further ideas how to debug this are greatly appreciated.

On Wed, Jul 20, 2016 at 7:13 PM, Juho Mäkinen <>

> I just recently upgraded our cluster to 2.2.7 and after turning the
> cluster under production load the instances started to show high load (as
> shown by uptime) without any apparent reason and I'm not quite sure what
> could be causing it.
> We are running on i2.4xlarge, so we have 16 cores, 120GB of ram, four
> 800GB SSDs (set as lvm stripe into one big lvol). Running 3.13.0-87-generic
> on HVM virtualisation. Cluster has 26 TiB of data stored in two tables.
> Symptoms:
>  - High load, sometimes up to 30 for a short duration of few minutes, then
> the load drops back to the cluster average: 3-4
>  - Instances might have one compaction running, but might not have any
> compactions.
>  - Each node is serving around 250-300 reads per second and around 200
> writes per second.
>  - Restarting node fixes the problem for around 18-24 hours.
>  - No or very little IO-wait.
>  - top shows that around 3-10 threads are running on high cpu, but that
> alone should not cause a load of 20-30.
>  - Doesn't seem to be GC load: A system starts to show symptoms so that it
> has ran only one CMS sweep. Not like it would do constant stop-the-world
> gc's.
>  - top shows that the C* processes use 100G of RSS memory. I assume that
> this is because cassandra opens all SSTables with mmap() so that they will
> pop up in the RSS count because of this.
> What I've done so far:
>  - Rolling restart. Helped for about one day.
>  - Tried doing manual GC to the cluster.
>  - Increased heap from 8 GiB with CMS to 16 GiB with G1GC.
>  - sjk-plus shows bunch of SharedPool workers. Not sure what to make of
> this.
>  - Browsed over
> but
> didn't find any apparent
> I know that the general symptom of "system shows high load" is not very
> good and informative, but I don't know how to better describe what's going
> on. I appreciate all ideas what to try and how to debug this further.
>  - Garo

View raw message