incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurpreet Singh <gurpreet.si...@gmail.com>
Subject Re: kswapd0 causing read timeouts
Date Fri, 08 Jun 2012 09:00:39 GMT
Thanks Ruslan.
I will try the mmap_index_only.
Is there any guideline as to when to leave it to auto and when to use
mmap_index_only?

/G

On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov <ruslan.usifov@gmail.com>wrote:

> disk_access_mode: mmap??
>
> set to disk_access_mode: mmap_index_only in cassandra yaml
>
> 2012/6/8 Gurpreet Singh <gurpreet.singh@gmail.com>:
> > Hi,
> > I am testing cassandra 1.1 on a 1 node cluster.
> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
> >
> > cassandra 1.1.1
> > heap size: 8 gigs
> > key cache size in mb: 800 (used only 200mb till now)
> > memtable_total_space_in_mb : 2048
> >
> > I am running a read workload.. about 30 reads/second. no writes at all.
> > The system runs fine for roughly 12 hours.
> >
> > jconsole shows that my heap size has hardly touched 4 gigs.
> > top shows -
> >   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
> >   RES increases slowly from 6 gigs all the way to 15 gigs
> >   buffers are at a healthy 25 mb at some point and that goes down to 2
> mb in
> > these 12 hrs
> >   VIRT stays at 85 gigs
> >
> > I understand that SHR goes up because of mmap, RES goes up because it is
> > showing SHR value as well.
> >
> > After around 10-12 hrs, the cpu utilization of the system starts
> increasing,
> > and i notice that kswapd0 process starts becoming more active. Gradually,
> > the system cpu becomes high almost 70%, and the client starts getting
> > continuous timeouts. The fact that the buffers went down from 20 mb to 2
> mb
> > suggests that kswapd0 is probably swapping out the pagecache.
> >
> > Is there a way out of this to avoid the kswapd0 starting to do things
> even
> > when there is no swap configured?
> > This is very easily reproducible for me, and would like a way out of this
> > situation. Do i need to adjust vm memory management stuff like pagecache,
> > vfs_cache_pressure.. things like that?
> >
> > just some extra information, jna is installed, mlockall is successful.
> there
> > is no compaction running.
> > would appreciate any help on this.
> > Thanks
> > Gurpreet
> >
> >
>

Mime
View raw message