cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Lacefield <jlacefi...@datastax.com>
Subject Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Date Tue, 25 Mar 2014 20:47:15 GMT
Sorry to hear about the frustration.  How often are you deleting data/what
are you setting for ttl on cols?

Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
<http://www.linkedin.com/in/jlacefield>


<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>


On Tue, Mar 25, 2014 at 4:22 PM, Oleg Dulin <oleg.dulin@gmail.com> wrote:

> Sigh, so I am back to where I started from...
>
> I did lower gc_grace...
>
> jmap -histo:live shows heap is stuffed with DeletedColumn and
> ExpiringColumn....
>
> This is extremely frustrating.
>
>
> On 2014-03-11 19:24:50 +0000, Oleg Dulin said:
>
>  Good news is that since I lowered gc_grace period it collected over
>> 100Gigs of tombstones and seems much happier now.
>>
>> Oleg
>>
>> On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:
>>
>>  Hello,
>>>
>>>   You have several options:
>>>
>>>   1) going forward lower gc_grace_seconds http://www.datastax.com/
>>> documentation/cassandra/1.2/cassandra/configuration/
>>> configStorage_r.html?pagename=docs&version=1.2&file=
>>> configuration/storage_configuration#gc-grace-seconds
>>>        - this is very use case specific.  Default is 10 days.  Some
>>> users will put this at 0 for specific use cases.
>>>   2) you could also lower tombstone compaction threshold and interval to
>>> get tombstone compaction to fire more often on your tables/cfs:
>>> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
>>>   3) to clean out old tombstones you could always run a manual
>>> compaction, those these aren't typically recommended though:
>>> http://www.datastax.com/documentation/cassandra/1.2/
>>> cassandra/tools/toolsNodetool_r.html
>>>
>>>   For 1 and 2, be sure your disks can keep up with compaction to ensure
>>> tombstone, or other, compaction fires regularly enough to clean out old
>>> tombstones.  Also, you probably want to ensure you are using Level
>>> Compaction:  http://www.datastax.com/dev/blog/when-to-use-leveled-
>>> compaction.  Again, this assumes your disk system can handle the
>>> increased io from Leveled Compaction.
>>>
>>>   Also, you may be running into this with the older version of
>>> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
>>>
>>>   Hope this helps.
>>>
>>> Jonathan
>>>
>>>
>>> Jonathan Lacefield
>>> Solutions Architect, DataStax
>>> (404) 822 3487
>>> <image>
>>>
>>>
>>> <image>
>>>
>>>
>>> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <oleg.dulin@gmail.com>
>>> wrote:
>>> I get that :)
>>>
>>> What I'd like to know is how to fix that :)
>>>
>>> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>>>
>>> You have millions of org.apache.cassandra.db.DeletedColumn instances on
>>> the snapshot.
>>>
>>> This means you have lots of column tombstones, and I guess, which are
>>> read into memory by slice query.
>>>
>>>
>>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <oleg.dulin@gmail.com>
>>> wrote:
>>> I am trying to understand why one of my nodes keeps full GC.
>>>
>>> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>>>
>>> Consider the top entries from jmap -histo:live @
>>> http://pastebin.com/UaatHfpJ
>>>
>>> --
>>> Regards,
>>> Oleg Dulin
>>> http://www.olegdulin.com
>>>
>>>
>>> --
>>> Regards,
>>> Oleg Dulin
>>> http://www.olegdulin.com
>>>
>>
>> S
>>
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>

Mime
View raw message