incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurpreet Singh <gurpreet.si...@gmail.com>
Subject kswapd0 causing read timeouts
Date Fri, 08 Jun 2012 08:17:15 GMT
Hi,
I am testing cassandra 1.1 on a 1 node cluster.
8 core, 16 gb ram, 6 data disks raid0, no swap configured

cassandra 1.1.1
heap size: 8 gigs
key cache size in mb: 800 (used only 200mb till now)
memtable_total_space_in_mb : 2048

I am running a read workload.. about 30 reads/second. no writes at all.
The system runs fine for roughly 12 hours.

jconsole shows that my heap size has hardly touched 4 gigs.
top shows -
  SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
  RES increases slowly from 6 gigs all the way to 15 gigs
  buffers are at a healthy 25 mb at some point and that goes down to 2 mb
in these 12 hrs
  VIRT stays at 85 gigs

I understand that SHR goes up because of mmap, RES goes up because it is
showing SHR value as well.

After around 10-12 hrs, the cpu utilization of the system starts
increasing, and i notice that kswapd0 process starts becoming more active.
Gradually, the system cpu becomes high almost 70%, and the client starts
getting continuous timeouts. The fact that the buffers went down from 20 mb
to 2 mb suggests that kswapd0 is probably swapping out the pagecache.

Is there a way out of this to avoid the kswapd0 starting to do things even
when there is no swap configured?
This is very easily reproducible for me, and would like a way out of this
situation. Do i need to adjust vm memory management stuff like pagecache,
vfs_cache_pressure.. things like that?

just some extra information, jna is installed, mlockall is successful.
there is no compaction running.
would appreciate any help on this.
Thanks
Gurpreet

Mime
View raw message