cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ruslan usifov <ruslan.usi...@gmail.com>
Subject Re: kswapd0 causing read timeouts
Date Fri, 08 Jun 2012 18:59:10 GMT
2012/6/8 aaron morton <aaron@thelastpickle.com>:
> Ruslan,
> Why did you suggest changing the disk_access_mode ?

Because this bring problems on empty seat, in any case for me mmap
bring similar problem and i doesn't have find any solution to resolve
it, only  change disk_access_mode:-((. For me also will be interesting
hear results of author of this theme

>
> Gurpreet,
> I would leave the disk_access_mode with the default until you have a reason
> to change it.
>
>> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>
> is swap disabled ?
>
>> Gradually,
>> > the system cpu becomes high almost 70%, and the client starts getting
>> > continuous timeouts
>
> 70% of one core or 70% of all cores ?
> Check the server logs, is there GC activity ?
> check nodetool cfstats to see the read latency for the cf.
>
> Take a look at vmstat to see if you are swapping, and look at iostats to see
> if io is the problem
> http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/06/2012, at 9:00 PM, Gurpreet Singh wrote:
>
> Thanks Ruslan.
> I will try the mmap_index_only.
> Is there any guideline as to when to leave it to auto and when to use
> mmap_index_only?
>
> /G
>
> On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov <ruslan.usifov@gmail.com>
> wrote:
>>
>> disk_access_mode: mmap??
>>
>> set to disk_access_mode: mmap_index_only in cassandra yaml
>>
>> 2012/6/8 Gurpreet Singh <gurpreet.singh@gmail.com>:
>> > Hi,
>> > I am testing cassandra 1.1 on a 1 node cluster.
>> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
>> >
>> > cassandra 1.1.1
>> > heap size: 8 gigs
>> > key cache size in mb: 800 (used only 200mb till now)
>> > memtable_total_space_in_mb : 2048
>> >
>> > I am running a read workload.. about 30 reads/second. no writes at all.
>> > The system runs fine for roughly 12 hours.
>> >
>> > jconsole shows that my heap size has hardly touched 4 gigs.
>> > top shows -
>> >   SHR increasing slowly from 100 mb to 6.6 gigs in  these 12 hrs
>> >   RES increases slowly from 6 gigs all the way to 15 gigs
>> >   buffers are at a healthy 25 mb at some point and that goes down to 2
>> > mb in
>> > these 12 hrs
>> >   VIRT stays at 85 gigs
>> >
>> > I understand that SHR goes up because of mmap, RES goes up because it is
>> > showing SHR value as well.
>> >
>> > After around 10-12 hrs, the cpu utilization of the system starts
>> > increasing,
>> > and i notice that kswapd0 process starts becoming more active.
>> > Gradually,
>> > the system cpu becomes high almost 70%, and the client starts getting
>> > continuous timeouts. The fact that the buffers went down from 20 mb to 2
>> > mb
>> > suggests that kswapd0 is probably swapping out the pagecache.
>> >
>> > Is there a way out of this to avoid the kswapd0 starting to do things
>> > even
>> > when there is no swap configured?
>> > This is very easily reproducible for me, and would like a way out of
>> > this
>> > situation. Do i need to adjust vm memory management stuff like
>> > pagecache,
>> > vfs_cache_pressure.. things like that?
>> >
>> > just some extra information, jna is installed, mlockall is successful.
>> > there
>> > is no compaction running.
>> > would appreciate any help on this.
>> > Thanks
>> > Gurpreet
>> >
>> >
>
>
>

Mime
View raw message