cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Dulin <oleg.du...@gmail.com>
Subject Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Date Tue, 25 Mar 2014 20:22:37 GMT
Sigh, so I am back to where I started from...

I did lower gc_grace...

jmap -histo:live shows heap is stuffed with DeletedColumn and 
ExpiringColumn....

This is extremely frustrating.

On 2014-03-11 19:24:50 +0000, Oleg Dulin said:

> Good news is that since I lowered gc_grace period it collected over 
> 100Gigs of tombstones and seems much happier now.
> 
> Oleg
> 
> On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:
> 
>> Hello,
>> 
>>   You have several options:
>> 
>>   1) going forward lower gc_grace_seconds 
>> http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds

>> 
>>        - this is very use case specific.  Default is 10 days.  Some 
>> users will put this at 0 for specific use cases.
>>   2) you could also lower tombstone compaction threshold and interval 
>> to get tombstone compaction to fire more often on your tables/cfs:  
>> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
>>   3) to clean out old tombstones you could always run a manual 
>> compaction, those these aren't typically recommended though:  
>> http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html

>> 
>>        
>>   For 1 and 2, be sure your disks can keep up with compaction to ensure 
>> tombstone, or other, compaction fires regularly enough to clean out old 
>> tombstones.  Also, you probably want to ensure you are using Level 
>> Compaction:  
>> http://www.datastax.com/dev/blog/when-to-use-leveled-compaction.  
>> Again, this assumes your disk system can handle the increased io from 
>> Leveled Compaction.
>> 
>>   Also, you may be running into this with the older version of 
>> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
>> 
>>   Hope this helps.
>> 
>> Jonathan
>> 
>> 
>> Jonathan Lacefield
>> Solutions Architect, DataStax
>> (404) 822 3487
>> <image>
>> 
>> 
>> <image>
>> 
>> 
>> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
>> I get that :)
>> 
>> What I'd like to know is how to fix that :)
>> 
>> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>> 
>> You have millions of org.apache.cassandra.db.DeletedColumn instances on 
>> the snapshot.
>> 
>> This means you have lots of column tombstones, and I guess, which are 
>> read into memory by slice query. 
>> 
>> 
>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
>> I am trying to understand why one of my nodes keeps full GC.
>> 
>> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>> 
>> Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
>> 
>> --
>> Regards,
>> Oleg Dulin
>> http://www.olegdulin.com
>> 
>> 
>> --
>> Regards,
>> Oleg Dulin
>> http://www.olegdulin.com
> 
> S


-- 
Regards,
Oleg Dulin
http://www.olegdulin.com



Mime
View raw message