cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steinmaurer, Thomas" <thomas.steinmau...@dynatrace.com>
Subject nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
Date Mon, 28 May 2018 14:43:49 GMT
Hello,

on a quite capable machine with 32 physical cores (64 vCPUs) we see sporadic CPU usage up
to 50% caused by nodetool on this box, thus digged a bit further. A few observations:

1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we are running Cassandra
with e.g. Xmx31G, nodetool is started with Xmx31G as well
2) As -XX:ParallelGCThreads is not explicitly set upon startup, this basically defaults to
a value dependent on the number of cores. In our case, with the machine above, the number
of parallel GC threads for the JVM is set to 43!
3) Test-wise, we have adapted the nodetool startup script in a way to get a Java Flight Recording
file on JVM exit, thus with each nodetool invocation we can inspect a JFR file. Here we may
have seen System.gc() calls (without visible knowledge where they come from), GC times for
the entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both Xmx128M (default
as it seems) and Xmx31G

After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup script, CPU usage
spikes by nodetool are entirely gone.

Is this something which has been already adapted/tackled in Cassandra versions > 2.1 or
worth to be considered as some sort of RFC?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freist?dterstra?e 313

Mime
View raw message