cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: Instability and memory problems
Date Mon, 21 Jun 2010 07:30:03 GMT
> How much of your physical RAM is dedicatd to the JVM?
>
> I forgot to say that you probably should consider lowering it
> significantly (to be continued, getting off the subway...).

So, it occurred to be you reported a 16 GB maximum heap size. If that
is a substantial portion of your total physical RAM, that would
probably be the reason why you see swapping. The operating system will
tend to use some heuristics to figure out what to keep in memory, and
whether to evict pages from cache or actually swap out applications.
The greater the memory pressure the greater the chance that the
operating system will start swapping you out. Swappiness 0 won't help
in this case.

(For some reason it never occurred to me before, but the JVM should,
if it doesn't already, provide a command line argument to make it use
mlock()/mlockall() to lock data in memory...)

Anyways: The solution in this case, if your 16 GB heap size is a
significant portion of system memory, is probably not to force the OS
not to swap, but rather lowering the JVM heap size. Consider that all
memory used by the JVM will be used for its heap, instead of caching
disk I/O. I am spekulating now on what your settings are, but suppose:

(1) Your *actual* need for JVM heap size is roughly 1 GB (I picked
something random and low).
(2) Your cassandra is configured to not keep extreme amounts of data
cached in-JVM (thus causing the real need to be roughly 1 GB, among
other things).
(3) Your maximum heap size is 16 GB.

(3) means that garbage collection becomes more efficient, because you
have lots more memory than you need, if looked at in isolation. But if
you are wasting 15 GB of would-be disk caching on maintaining the
oversize JVM heap, you will instead completely kill performance.

Probably there are two extremes, and a spectrum in between:

(1) Tune cassandra and the JVM to gobble up a *lot* of memory and
cache data in the JVM/cassandra.
(2) Tune cassandra and the JVM to gobble up an appropriate amount of
memory, and leave as much as possible for the operating system cache
mechanism.

(2) is likely to be a lot more performant than (1), assuming it works
and you don't swap.

If you do have a significantly oversized heap, one potential resulting
behavior is overall poor performance due to lack of caching, and
periodically much worse performance in relation to swap storms during
GC:s.

-- 
/ Peter Schuller

Mime
View raw message