cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Pfau <matthias.p...@tutao.de>
Subject 2.2 eats memory
Date Mon, 27 Aug 2018 11:32:47 GMT
Hi there,
after upgrading from 2.1 to 2.2.13, Cassandra eats up all available memory within one week.
The following is a diagram of the left available RAM of a single node over the course of a
week:
https://imgur.com/a/H9BDBxC <https://imgur.com/a/H9BDBxC>

Nodes are bare metal, 12 cores with 64GB RAM each, swapping disabled and configured with row
cache disabled and auto sized key cache. Jemalloc is not configured.

This is the same for all nodes. There is no noticeable cpu load on these nodes, disk io is
also fine, just as tpstats. Heap dumps look fine and if you check memory usage via JMX everything
seems to be in order also (max heap 8GB, heap memory usage goes down to 1GB after GC)

The memory usage reported by top (res) is always around 20GB for the cassandra process which
also seems to high for us (heap + off heap) as the offheap memory allocated by DirectByteBuffers
is only around 400MB (reported by jxray).

The available system memory slowly degrades all the time without any other memory intensive
processes running. We currently drain and restart our nodes, if the available memory sinks
below 5GB. That means that after a week, there is an offset of around 40GB between the reported
memory usage of cassandra and the real memory usage.

If we stop cassandra, nearly 100% of the system memory is available again.

Has anybody observed similar memory usage with 2.2?

If so, how to measure cassandras real memory usage if top and other system tools are way off?

Do you know if this problem has been solved with 3.11?

Cheers,
Matthias

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message