incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sankalp kohli <kohlisank...@gmail.com>
Subject Re: Heap is not released and streaming hangs at 0%
Date Sat, 22 Jun 2013 01:17:15 GMT
I will take a heap dump and see whats in there rather than guessing.


On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot <btalbot@aeriagames.com>wrote:

> bloom_filter_fp_chance = 0.7 is probably way too large to be effective and
> you'll probably have issues compacting deleted rows and get poor read
> performance with a value that high.  I'd guess that anything larger than
> 0.1 might as well be 1.0.
>
> -Bryan
>
>
>
> On Fri, Jun 21, 2013 at 5:58 AM, srmore <comomore@gmail.com> wrote:
>
>>
>> On Fri, Jun 21, 2013 at 2:53 AM, aaron morton <aaron@thelastpickle.com>wrote:
>>
>>> > nodetool -h localhost flush didn't do much good.
>>>
>>> Do you have 100's of millions of rows ?
>>> If so see recent discussions about reducing the bloom_filter_fp_chance
>>> and index_sampling.
>>>
>> Yes, I have 100's of millions of rows.
>>
>>
>>>
>>> If this is an old schema you may be using the very old setting of
>>> 0.000744 which creates a lot of bloom filters.
>>>
>>> bloom_filter_fp_chance value that was changed from default to 0.1,
>> looked at the filters and they are about 2.5G on disk and I have around 8G
>> of heap.
>> I will try increasing the value to 0.7 and report my results.
>>
>> It also appears to be a case of hard GC failure (as Rob mentioned) as the
>> heap is never released, even after 24+ hours of idle time, the JVM needs to
>> be restarted to reclaim the heap.
>>
>> Cheers
>>>
>>>    -----------------
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 20/06/2013, at 6:36 AM, Wei Zhu <wz1975@yahoo.com> wrote:
>>>
>>> If you want, you can try to force the GC through Jconsole.
>>> Memory->Perform GC.
>>>
>>> It theoretically triggers a full GC and when it will happen depends on
>>> the JVM
>>>
>>> -Wei
>>>
>>> ------------------------------
>>> *From: *"Robert Coli" <rcoli@eventbrite.com>
>>> *To: *user@cassandra.apache.org
>>> *Sent: *Tuesday, June 18, 2013 10:43:13 AM
>>> *Subject: *Re: Heap is not released and streaming hangs at 0%
>>>
>>> On Tue, Jun 18, 2013 at 10:33 AM, srmore <comomore@gmail.com> wrote:
>>> > But then shouldn't JVM C G it eventually ? I can still see Cassandra
>>> alive
>>> > and kicking but looks like the heap is locked up even after the
>>> traffic is
>>> > long stopped.
>>>
>>> No, when GC system fails this hard it is often a permanent failure
>>> which requires a restart of the JVM.
>>>
>>> > nodetool -h localhost flush didn't do much good.
>>>
>>> This adds support to the idea that your heap is too full, and not full
>>> of memtables.
>>>
>>> You could try nodetool -h localhost invalidatekeycache, but that
>>> probably will not free enough memory to help you.
>>>
>>> =Rob
>>>
>>>
>>>
>>
>

Mime
View raw message