cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: How can I limit the non-heap memory for Cassandra
Date Fri, 18 Jan 2019 20:01:07 GMT
Hello Chris,

I must admit I am a bit confused about what you need exactly, I'll try to
do my best :).


> would like to place limits on it to avoid it becoming a “noisy neighbor”

But we also don’t want it killed by the oom killer, so just placing limits
> on the container won't help.


This sound contradictory to me. When the available memory is fully used and
the memory cannot be taken elsewhere the program cannot continue and OOM
cannot be avoided. So it's probably one thing or the other I would say
¯\_(ツ)_/¯.

Is there’s a way to limit Cassandra’s off-heap memory usage?


If we consider this perspective: "cointainerMemory = NativeMemory +
HeapMemory",
Then controlling the JVM heap memory and the container memory, you also
control the off-heap/native memory. So practically yes, you can set the
off-heap memory size, not directly, but by reducing the JVM heap size.
The option is in jvm.options (or cassandra-env.sh): MAX_HEAP_SIZE="X"  (it
comes with some tradeoff as well of course, if you're going this path I
recommend this post from Jon about JVM/GarbageCollection which is a tricky
piece of Cassandra operations
http://thelastpickle.com/blog/2018/04/11/gc-tuning.html)

You can also control each (most?) of the off-heap structures individually.
It's a bit split here and there between the distinct configuration files
and at the table level.
For example, if you're running out of Native Memory, you can maybe:

- Consider adding RAM or use a bigger instance type in the cloud.
- Reduce bloom filters ? -->
http://cassandra.apache.org/doc/latest/operating/bloom_filters.html?highlight=bloom_filter_fp_chance#bloom-filters
- Disable Row caches ? If you have troubles with memory, I would start
there probably (You did not give us your version of Cassandra though).
- Reduce the max_index_interval? -->
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableMax_index_interval
- ...

*Long Story :)*

 It's the C* operator job to ensure that the hardware choice and usage is
optimal or at least that the sum of the memory needed by off-heap
structures stays below what's available and does not produce any OOM. Some
structures are capped (like the key cache size) some other will grow with
the data (like bloom filters). Thus it's good to have some breathing room
for growth and to have a monitoring system in place (this is something I
advocate for at any occasion :D). Finding the right balance is part of the
job of many of us here around :).

That being said, it's rare we are fighting this kind of OOM because, in a
huge majority of the cluster, we rely strongly on page caching and we try
to have as much possible 'free' native memory for that purpose. We run into
problems way before running out of native memory in many cases.

Generally, a Cassandra cluster with the recommended (64 GB of RAM maybe?)
or at least decent (32 GB?) hardware and the default configuration should
hopefully work nicely. The schema design might make things worse and of
course, you can optimize and reduce the cost, sometimes in a
substantial way. But globally Cassandra and the default configuration give
a good starting point I think.

One last thing is that the more details you share, the sharper and accurate
our answers can be. I feel like I told you *everything* I know about memory
because it wasn't clear to me what you needed :). Specifying the Cassandra
version, and telling something about your specific case like the memory
total size or the JVM configuration would probably induce (faster/better)
responses from us :).

I hope this still helps.

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com





Le jeu. 3 janv. 2019 à 00:03, Chris Mildebrandt <chris@octarinesec.com> a
écrit :

> Hi,
>
> Is there’s a way to limit Cassandra’s off-heap memory usage? I can’t find
> a way to limit the memory used for row caches, bloom filters, etc. We’re
> running Cassandra in a container and would like to place limits on it to
> avoid it becoming a “noisy neighbor”. But we also don’t want it killed by
> the oom killer, so just placing limits on the container won't help.
>
> Thanks,
> -Chris
>

Mime
View raw message