cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ruslan usifov <ruslan.usi...@gmail.com>
Subject Re: kswapd0 causing read timeouts
Date Fri, 08 Jun 2012 08:21:12 GMT
disk_access_mode: mmap??

set to disk_access_mode: mmap_index_only in cassandra yaml

2012/6/8 Gurpreet Singh <gurpreet.singh@gmail.com>:
> Hi,
> I am testing cassandra 1.1 on a 1 node cluster.
> 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>
> cassandra 1.1.1
> heap size: 8 gigs
> key cache size in mb: 800 (used only 200mb till now)
> memtable_total_space_in_mb : 2048
>
> I am running a read workload.. about 30 reads/second. no writes at all.
> The system runs fine for roughly 12 hours.
>
> jconsole shows that my heap size has hardly touched 4 gigs.
> top shows -
>   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
>   RES increases slowly from 6 gigs all the way to 15 gigs
>   buffers are at a healthy 25 mb at some point and that goes down to 2 mb in
> these 12 hrs
>   VIRT stays at 85 gigs
>
> I understand that SHR goes up because of mmap, RES goes up because it is
> showing SHR value as well.
>
> After around 10-12 hrs, the cpu utilization of the system starts increasing,
> and i notice that kswapd0 process starts becoming more active. Gradually,
> the system cpu becomes high almost 70%, and the client starts getting
> continuous timeouts. The fact that the buffers went down from 20 mb to 2 mb
> suggests that kswapd0 is probably swapping out the pagecache.
>
> Is there a way out of this to avoid the kswapd0 starting to do things even
> when there is no swap configured?
> This is very easily reproducible for me, and would like a way out of this
> situation. Do i need to adjust vm memory management stuff like pagecache,
> vfs_cache_pressure.. things like that?
>
> just some extra information, jna is installed, mlockall is successful. there
> is no compaction running.
> would appreciate any help on this.
> Thanks
> Gurpreet
>
>

Mime
View raw message