incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Holger Hoffstaette" <holger.hoffstae...@googlemail.com>
Subject Re: kswapd0 causing read timeouts
Date Mon, 18 Jun 2012 12:46:14 GMT
On Thu, 14 Jun 2012 11:08:39 -0700, Gurpreet Singh wrote:

> JNA is installed. swappiness was 0. vfs_cache_pressure was 100.
[..]
> Right now, i am handling only 80 gigs of data. kernel version is 2.6.26.
> java version is 1.6.21

- your JVM is way too old and will leak native memory when you use JMX.
Use either latest JDK6 or try 7u5, works for me.

- your kernel is way too old and suffers from kswapd going crazy
*especially* when swappiness=0, which *does not do what you think it
does*. Upgrade your kernel (latest stable 3.3.x is good, 3.4.3 works for
me too). This has nothing to do with Cassandra, other people have found
the same problem (esp. with old RH "Enterprise" kernels).

- don't set swappiness=0 unless you add a kernel patch (to 3.4.x) that
will be part of 3.5 (http://t.co/l0XyLkCg) and which will have the effect
that people expect. This works *really* great, and swapping only occurs
when absolutely necessary - e.g. when an app is leaking memory.

- vfs_cache_pressure has no play in any of this.

- use sysctl to check that vm.dirty_background_ratio and vm.dirty_ratio
are in the low single-digits (like say 2 & 4). This will lead to less
bursty writeout activity.

Cassandra's (current) use of mmap is far from optimal, but when the JVM
and kernel are not doing what they are supposed to do, it can become
especially "harmful". Just look at MongoDB.

-h



Mime
View raw message