Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5C55510224 for ; Tue, 25 Mar 2014 20:47:53 +0000 (UTC) Received: (qmail 35169 invoked by uid 500); 25 Mar 2014 20:47:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 34992 invoked by uid 500); 25 Mar 2014 20:47:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 34922 invoked by uid 99); 25 Mar 2014 20:47:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2014 20:47:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jlacefield@datastax.com designates 209.85.128.175 as permitted sender) Received: from [209.85.128.175] (HELO mail-ve0-f175.google.com) (209.85.128.175) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2014 20:47:38 +0000 Received: by mail-ve0-f175.google.com with SMTP id oz11so1247562veb.20 for ; Tue, 25 Mar 2014 13:47:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=tjaFox9ffr5AFStTTgu2CMR4dj+FyMhGgkHicrCGKko=; b=IN5Mmij7V7LqYHx5K94Iq/uyq4ZDoin/Jz2c4SCk+dLBHxkSuLr8kX0FcyLQxKAqsy FolMMePrVRMqJIhZ7YG7+64iXMrafHYJlgSHqp8QvzGFhWI8PxwJIJ8yK9mPGMSz4kFb p6SIecnsWx72lHTeOShqOWb8+axkwcAGQ0u9GeopgCkJ3fSN6IzlDu0uSp9BOjW2mzGY 4PWUDEOppwyASvdTIr0WtAM5PxlvV/yyUKOh14xFokezRhudfwBSgJlqC4Akc2pOvFvh 0Vc9HtElUS74mCMkRYPhkb4kGmxeBmLs/DpXwkVppFK9tbF4ecxQpiHBO/oisuQ4M5JF eUcA== X-Gm-Message-State: ALoCoQlbmC0SAK0TRNHGxb3tlLiX+/+FijT3A2i4AI9hRA+WZTUzCGjd3UYKluukV1IDJ6f8+adP MIME-Version: 1.0 X-Received: by 10.221.61.210 with SMTP id wx18mr2883684vcb.27.1395780435626; Tue, 25 Mar 2014 13:47:15 -0700 (PDT) Received: by 10.52.183.34 with HTTP; Tue, 25 Mar 2014 13:47:15 -0700 (PDT) In-Reply-To: References: Date: Tue, 25 Mar 2014 16:47:15 -0400 Message-ID: Subject: Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram From: Jonathan Lacefield To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=001a1134a86e0ce19d04f5747566 X-Virus-Checked: Checked by ClamAV on apache.org --001a1134a86e0ce19d04f5747566 Content-Type: text/plain; charset=ISO-8859-1 Sorry to hear about the frustration. How often are you deleting data/what are you setting for ttl on cols? Jonathan Lacefield Solutions Architect, DataStax (404) 822 3487 On Tue, Mar 25, 2014 at 4:22 PM, Oleg Dulin wrote: > Sigh, so I am back to where I started from... > > I did lower gc_grace... > > jmap -histo:live shows heap is stuffed with DeletedColumn and > ExpiringColumn.... > > This is extremely frustrating. > > > On 2014-03-11 19:24:50 +0000, Oleg Dulin said: > > Good news is that since I lowered gc_grace period it collected over >> 100Gigs of tombstones and seems much happier now. >> >> Oleg >> >> On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said: >> >> Hello, >>> >>> You have several options: >>> >>> 1) going forward lower gc_grace_seconds http://www.datastax.com/ >>> documentation/cassandra/1.2/cassandra/configuration/ >>> configStorage_r.html?pagename=docs&version=1.2&file= >>> configuration/storage_configuration#gc-grace-seconds >>> - this is very use case specific. Default is 10 days. Some >>> users will put this at 0 for specific use cases. >>> 2) you could also lower tombstone compaction threshold and interval to >>> get tombstone compaction to fire more often on your tables/cfs: >>> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436 >>> 3) to clean out old tombstones you could always run a manual >>> compaction, those these aren't typically recommended though: >>> http://www.datastax.com/documentation/cassandra/1.2/ >>> cassandra/tools/toolsNodetool_r.html >>> >>> For 1 and 2, be sure your disks can keep up with compaction to ensure >>> tombstone, or other, compaction fires regularly enough to clean out old >>> tombstones. Also, you probably want to ensure you are using Level >>> Compaction: http://www.datastax.com/dev/blog/when-to-use-leveled- >>> compaction. Again, this assumes your disk system can handle the >>> increased io from Leveled Compaction. >>> >>> Also, you may be running into this with the older version of >>> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541 >>> >>> Hope this helps. >>> >>> Jonathan >>> >>> >>> Jonathan Lacefield >>> Solutions Architect, DataStax >>> (404) 822 3487 >>> >>> >>> >>> >>> >>> >>> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin >>> wrote: >>> I get that :) >>> >>> What I'd like to know is how to fix that :) >>> >>> On 2014-03-09 20:24:54 +0000, Takenori Sato said: >>> >>> You have millions of org.apache.cassandra.db.DeletedColumn instances on >>> the snapshot. >>> >>> This means you have lots of column tombstones, and I guess, which are >>> read into memory by slice query. >>> >>> >>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin >>> wrote: >>> I am trying to understand why one of my nodes keeps full GC. >>> >>> I have Xmx set to 8gigs, memtable total size is 2 gigs. >>> >>> Consider the top entries from jmap -histo:live @ >>> http://pastebin.com/UaatHfpJ >>> >>> -- >>> Regards, >>> Oleg Dulin >>> http://www.olegdulin.com >>> >>> >>> -- >>> Regards, >>> Oleg Dulin >>> http://www.olegdulin.com >>> >> >> S >> > > > -- > Regards, > Oleg Dulin > http://www.olegdulin.com > > > --001a1134a86e0ce19d04f5747566 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Sorry to hear about the frustration. =A0How often are you = deleting data/what are you setting for ttl on cols? =A0

Jonathan LacefieldSolutions Architect, DataStax
(404) 822 3487






On Tue, Mar 25, 2014 at 4:22 PM, Oleg Du= lin <oleg.dulin@gmail.com> wrote:
Sigh, so I am back to where I started from...

I did lower gc_grace...

jmap -histo:live shows heap is stuffed with DeletedColumn and ExpiringColum= n....

This is extremely frustrating.


On 2014-03-11 19:24:50 +0000, Oleg Dulin said:

Good news is that since I lowered gc_grace period it collected over 100Gigs= of tombstones and seems much happier now.

Oleg

On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:

Hello,

=A0 You have several options:

=A0 1) going forward lower gc_grace_seconds http://www.datastax.com/= documentation/cassandra/1.2/cassandra/configuration/configSto= rage_r.html?pagename=3Ddocs&version=3D1.2&file=3Dconf= iguration/storage_configuration#gc-grace-seconds
=A0 =A0 =A0 =A0- this is very use case specific. =A0Default is 10 days. =A0= Some users will put this at 0 for specific use cases.
=A0 2) you could also lower tombstone compaction threshold and interval to = get tombstone compaction to fire more often on your tables/cfs: =A0https://datastax.jira.com/wiki/pages/viewpage.act= ion?pageId=3D54493436
=A0 3) to clean out old tombstones you could always run a manual compaction= , those these aren't typically recommended though: =A0http://www.datastax.com/documentation/cas= sandra/1.2/cassandra/tools/toolsNodetool_r.html
=A0 =A0 =A0 =A0
=A0 For 1 and 2, be sure your disks can keep up with compaction to ensure t= ombstone, or other, compaction fires regularly enough to clean out old tomb= stones. =A0Also, you probably want to ensure you are using Level Compaction= : =A0http://www.datastax.com/dev/blog/when-to-use= -leveled-compaction. =A0Again, this assumes your disk system can= handle the increased io from Leveled Compaction.

=A0 Also, you may be running into this with the older version of Cassandra:= =A0https://issues.apache.org/jira/browse/CASSANDRA-6= 541

=A0 Hope this helps.

Jonathan


Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
<image>


<image>


On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
I get that :)

What I'd like to know is how to fix that :)

On 2014-03-09 20:24:54 +0000, Takenori Sato said:

You have millions of=A0org.apache.cassandra.db.DeletedColumn=A0insta= nces on the snapshot.

This means you have lots of column tombstones, and I guess, which are read = into memory by slice query.=A0


On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
I am trying to understand why one of my nodes keeps full GC.

I have Xmx set to 8gigs, memtable total size is 2 gigs.

Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ

--
Regards,
Oleg Dulin
http://www.olegdulin= .com


--
Regards,
Oleg Dulin
http://www.olegdulin= .com

S


--
Regards,
Oleg Dulin
http://www.olegdulin= .com



--001a1134a86e0ce19d04f5747566--