incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Golick <jamesgol...@gmail.com>
Subject Re: Instability and memory problems
Date Mon, 21 Jun 2010 15:07:32 GMT
Just an update here. We're now entirely on standard IO mode, and everything
is stable and happy. There hasn't been much of a performance hit, if at all.

- James

On Mon, Jun 21, 2010 at 3:30 AM, Peter Schuller <peter.schuller@infidyne.com
> wrote:

> > How much of your physical RAM is dedicatd to the JVM?
> >
> > I forgot to say that you probably should consider lowering it
> > significantly (to be continued, getting off the subway...).
>
> So, it occurred to be you reported a 16 GB maximum heap size. If that
> is a substantial portion of your total physical RAM, that would
> probably be the reason why you see swapping. The operating system will
> tend to use some heuristics to figure out what to keep in memory, and
> whether to evict pages from cache or actually swap out applications.
> The greater the memory pressure the greater the chance that the
> operating system will start swapping you out. Swappiness 0 won't help
> in this case.
>
> (For some reason it never occurred to me before, but the JVM should,
> if it doesn't already, provide a command line argument to make it use
> mlock()/mlockall() to lock data in memory...)
>
> Anyways: The solution in this case, if your 16 GB heap size is a
> significant portion of system memory, is probably not to force the OS
> not to swap, but rather lowering the JVM heap size. Consider that all
> memory used by the JVM will be used for its heap, instead of caching
> disk I/O. I am spekulating now on what your settings are, but suppose:
>
> (1) Your *actual* need for JVM heap size is roughly 1 GB (I picked
> something random and low).
> (2) Your cassandra is configured to not keep extreme amounts of data
> cached in-JVM (thus causing the real need to be roughly 1 GB, among
> other things).
> (3) Your maximum heap size is 16 GB.
>
> (3) means that garbage collection becomes more efficient, because you
> have lots more memory than you need, if looked at in isolation. But if
> you are wasting 15 GB of would-be disk caching on maintaining the
> oversize JVM heap, you will instead completely kill performance.
>
> Probably there are two extremes, and a spectrum in between:
>
> (1) Tune cassandra and the JVM to gobble up a *lot* of memory and
> cache data in the JVM/cassandra.
> (2) Tune cassandra and the JVM to gobble up an appropriate amount of
> memory, and leave as much as possible for the operating system cache
> mechanism.
>
> (2) is likely to be a lot more performant than (1), assuming it works
> and you don't swap.
>
> If you do have a significantly oversized heap, one potential resulting
> behavior is overall poor performance due to lack of caching, and
> periodically much worse performance in relation to swap storms during
> GC:s.
>
> --
> / Peter Schuller
>

Mime
View raw message