incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Heap is not released and streaming hangs at 0%
Date Fri, 28 Jun 2013 04:58:54 GMT
>  I do not see the out of heap errors but I am taking a bit of a performance hit.
Take a look at nodetool cfhistograms to see how many SSTables are being touched per read and
the local read latency. 

In general if you are hitting more than 4 it's not great.

>  BloomFilterFalseRatio is 0.8367977262013025 this was the reason behind bumping bloom_filter_fp_chance.
Not sure I understand the logic there. 


> 
> I also encountered similar problem. I dump the jvm heap and analyse it by eclipse mat.
The eclipse plugin told me there are 10334 instances of SSTableReader, consuming 6.6G memory.
I found the CompactionExecutor thread held  8000+ SSTalbeReader object. I wonder why there
are so many SSTableReader in memory. 
I would guess you are using Levelled compaction strategy with the default 5MB file size. 
When analysing the heap, if bloom filters are taking up too much memory they will show up
as long[][] 

Cheers


-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 27/06/2013, at 2:36 AM, srmore <comomore@gmail.com> wrote:

> On Wed, Jun 26, 2013 at 12:16 AM, aaron morton <aaron@thelastpickle.com> wrote:
>> bloom_filter_fp_chance value that was changed from default to 0.1, looked at the
filters and they are about 2.5G on disk and I have around 8G of heap.
>> I will try increasing the value to 0.7 and report my results. 
> You need to re-write the sstables on disk using nodetool upgradesstables. Otherwise only
the new tables with have the 0.1 setting. 
> 
>> I will try increasing the value to 0.7 and report my results. 
> No need to, it will probably be something like "Oh no, really, what, how, please make
it stop" :)
> 0.7 will mean reads will hit most / all of the SSTables for the CF. 
> 
> Changing the bloom_filter_fp_chance to 0.7 did seem to correct the problem in short run.
I do not see the out of heap errors but I am taking a bit of a performance hit. Planning to
run some more tests, also  my BloomFilterFalseRatio is 0.8367977262013025 this was the reason
behind bumping bloom_filter_fp_chance.
>  
> 
> I covered a high row situation in on of my talks at the summit this month, the slide
deck is here http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass
and the videos will soon be up at Planet Cassandra. 
> 
> This was/is extremely helpful Aaron, cannot thank you enough for sharing this with the
community, eagerly looking forward for the video.
> 
> Rebuild the sstables, then reduce the index_interval if you still need to reduce mem
pressure. 
>  
> Cheers
> 
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> 
> On 22/06/2013, at 1:17 PM, sankalp kohli <kohlisankalp@gmail.com> wrote:
> 
>> I will take a heap dump and see whats in there rather than guessing. 
>> 
>> 
>> On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot <btalbot@aeriagames.com> wrote:
>> bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll
probably have issues compacting deleted rows and get poor read performance with a value that
high.  I'd guess that anything larger than 0.1 might as well be 1.0.
>> 
>> -Bryan
>> 
>> 
>> 
>> On Fri, Jun 21, 2013 at 5:58 AM, srmore <comomore@gmail.com> wrote:
>> 
>> On Fri, Jun 21, 2013 at 2:53 AM, aaron morton <aaron@thelastpickle.com> wrote:
>>> > nodetool -h localhost flush didn't do much good.
>> Do you have 100's of millions of rows ?
>> If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling.

>> Yes, I have 100's of millions of rows. 
>>  
>> 
>> If this is an old schema you may be using the very old setting of 0.000744 which
creates a lot of bloom filters. 
>> 
>> bloom_filter_fp_chance value that was changed from default to 0.1, looked at the
filters and they are about 2.5G on disk and I have around 8G of heap.
>> I will try increasing the value to 0.7 and report my results. 
>> 
>> It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is
never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim
the heap.
>> 
>> Cheers
>>  
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 20/06/2013, at 6:36 AM, Wei Zhu <wz1975@yahoo.com> wrote:
>> 
>>> If you want, you can try to force the GC through Jconsole. Memory->Perform
GC.
>>> 
>>> It theoretically triggers a full GC and when it will happen depends on the JVM
>>> 
>>> -Wei
>>> 
>>> From: "Robert Coli" <rcoli@eventbrite.com>
>>> To: user@cassandra.apache.org
>>> Sent: Tuesday, June 18, 2013 10:43:13 AM
>>> Subject: Re: Heap is not released and streaming hangs at 0%
>>> 
>>> On Tue, Jun 18, 2013 at 10:33 AM, srmore <comomore@gmail.com> wrote:
>>> > But then shouldn't JVM C G it eventually ? I can still see Cassandra alive
>>> > and kicking but looks like the heap is locked up even after the traffic
is
>>> > long stopped.
>>> 
>>> No, when GC system fails this hard it is often a permanent failure
>>> which requires a restart of the JVM.
>>> 
>>> > nodetool -h localhost flush didn't do much good.
>>> 
>>> This adds support to the idea that your heap is too full, and not full
>>> of memtables.
>>> 
>>> You could try nodetool -h localhost invalidatekeycache, but that
>>> probably will not free enough memory to help you.
>>> 
>>> =Rob
>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message