incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Heap is not released and streaming hangs at 0%
Date Wed, 26 Jun 2013 05:16:20 GMT
> bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters
and they are about 2.5G on disk and I have around 8G of heap.
> I will try increasing the value to 0.7 and report my results. 
You need to re-write the sstables on disk using nodetool upgradesstables. Otherwise only the
new tables with have the 0.1 setting. 

> I will try increasing the value to 0.7 and report my results. 
No need to, it will probably be something like "Oh no, really, what, how, please make it stop"
:)
0.7 will mean reads will hit most / all of the SSTables for the CF. 

I covered a high row situation in on of my talks at the summit this month, the slide deck
is here http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass
and the videos will soon be up at Planet Cassandra. 

Rebuild the sstables, then reduce the index_interval if you still need to reduce mem pressure.

 
Cheers


-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com


On 22/06/2013, at 1:17 PM, sankalp kohli <kohlisankalp@gmail.com> wrote:

> I will take a heap dump and see whats in there rather than guessing. 
> 
> 
> On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot <btalbot@aeriagames.com> wrote:
> bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably
have issues compacting deleted rows and get poor read performance with a value that high.
 I'd guess that anything larger than 0.1 might as well be 1.0.
> 
> -Bryan
> 
> 
> 
> On Fri, Jun 21, 2013 at 5:58 AM, srmore <comomore@gmail.com> wrote:
> 
> On Fri, Jun 21, 2013 at 2:53 AM, aaron morton <aaron@thelastpickle.com> wrote:
>> > nodetool -h localhost flush didn't do much good.
> Do you have 100's of millions of rows ?
> If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling.

> Yes, I have 100's of millions of rows. 
>  
> 
> If this is an old schema you may be using the very old setting of 0.000744 which creates
a lot of bloom filters. 
> 
> bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters
and they are about 2.5G on disk and I have around 8G of heap.
> I will try increasing the value to 0.7 and report my results. 
> 
> It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never
released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the
heap.
> 
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 20/06/2013, at 6:36 AM, Wei Zhu <wz1975@yahoo.com> wrote:
> 
>> If you want, you can try to force the GC through Jconsole. Memory->Perform GC.
>> 
>> It theoretically triggers a full GC and when it will happen depends on the JVM
>> 
>> -Wei
>> 
>> From: "Robert Coli" <rcoli@eventbrite.com>
>> To: user@cassandra.apache.org
>> Sent: Tuesday, June 18, 2013 10:43:13 AM
>> Subject: Re: Heap is not released and streaming hangs at 0%
>> 
>> On Tue, Jun 18, 2013 at 10:33 AM, srmore <comomore@gmail.com> wrote:
>> > But then shouldn't JVM C G it eventually ? I can still see Cassandra alive
>> > and kicking but looks like the heap is locked up even after the traffic is
>> > long stopped.
>> 
>> No, when GC system fails this hard it is often a permanent failure
>> which requires a restart of the JVM.
>> 
>> > nodetool -h localhost flush didn't do much good.
>> 
>> This adds support to the idea that your heap is too full, and not full
>> of memtables.
>> 
>> You could try nodetool -h localhost invalidatekeycache, but that
>> probably will not free enough memory to help you.
>> 
>> =Rob
> 
> 
> 
> 


Mime
View raw message