Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4EDC410C00 for ; Sat, 22 Jun 2013 01:18:24 +0000 (UTC) Received: (qmail 55499 invoked by uid 500); 22 Jun 2013 01:18:21 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 55480 invoked by uid 500); 22 Jun 2013 01:18:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 55472 invoked by uid 99); 22 Jun 2013 01:18:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jun 2013 01:18:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kohlisankalp@gmail.com designates 209.85.216.178 as permitted sender) Received: from [209.85.216.178] (HELO mail-qc0-f178.google.com) (209.85.216.178) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jun 2013 01:18:16 +0000 Received: by mail-qc0-f178.google.com with SMTP id c11so5115490qcv.37 for ; Fri, 21 Jun 2013 18:17:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=5jBKuuNI0jVBC3L2dqHZy0mLiaDW80WYBU6vLgFpLv8=; b=pKRXdzvtYsl9g/v8vm8DTX1qS19vgRGsODU1h4MlCfnVLJK1vmS1U3qf59Uxgt8Mhm Ffn4zIgslldSKTlkzShpEQsFyrB8UEon0+uGbGlf7vU1f6uvTon41OZCDGV4BSG+3wrh 3IiXbZ7QshtQPHHuccuf1KOSNw3wMPfWBJD8VOwscK1MTmF2idGE8UkI9P5Q2X7avt79 lO0Q4EJc003Gm6ieCbiPR1c1qtYXGbKaLyt/U/LcUqL5tdpIqiUF1e1lodhHdEEunP8l HT2av942UgPyCYIlxHaPAN6YSTL+0bQxfUHU7o17CreTC5ZAQ4M48e7fgu1ivM73uNmP KfQg== X-Received: by 10.224.106.195 with SMTP id y3mr16660301qao.35.1371863875526; Fri, 21 Jun 2013 18:17:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.49.0.171 with HTTP; Fri, 21 Jun 2013 18:17:15 -0700 (PDT) In-Reply-To: References: <1371666998.53260.GenericBBA@web160905.mail.bf1.yahoo.com> <5F6EA5DA-FF40-49F4-A672-E59EC859AC0C@thelastpickle.com> From: sankalp kohli Date: Fri, 21 Jun 2013 18:17:15 -0700 Message-ID: Subject: Re: Heap is not released and streaming hangs at 0% To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf3074b33afb27ce04dfb3f2a7 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3074b33afb27ce04dfb3f2a7 Content-Type: text/plain; charset=ISO-8859-1 I will take a heap dump and see whats in there rather than guessing. On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot wrote: > bloom_filter_fp_chance = 0.7 is probably way too large to be effective and > you'll probably have issues compacting deleted rows and get poor read > performance with a value that high. I'd guess that anything larger than > 0.1 might as well be 1.0. > > -Bryan > > > > On Fri, Jun 21, 2013 at 5:58 AM, srmore wrote: > >> >> On Fri, Jun 21, 2013 at 2:53 AM, aaron morton wrote: >> >>> > nodetool -h localhost flush didn't do much good. >>> >>> Do you have 100's of millions of rows ? >>> If so see recent discussions about reducing the bloom_filter_fp_chance >>> and index_sampling. >>> >> Yes, I have 100's of millions of rows. >> >> >>> >>> If this is an old schema you may be using the very old setting of >>> 0.000744 which creates a lot of bloom filters. >>> >>> bloom_filter_fp_chance value that was changed from default to 0.1, >> looked at the filters and they are about 2.5G on disk and I have around 8G >> of heap. >> I will try increasing the value to 0.7 and report my results. >> >> It also appears to be a case of hard GC failure (as Rob mentioned) as the >> heap is never released, even after 24+ hours of idle time, the JVM needs to >> be restarted to reclaim the heap. >> >> Cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Consultant >>> New Zealand >>> >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 20/06/2013, at 6:36 AM, Wei Zhu wrote: >>> >>> If you want, you can try to force the GC through Jconsole. >>> Memory->Perform GC. >>> >>> It theoretically triggers a full GC and when it will happen depends on >>> the JVM >>> >>> -Wei >>> >>> ------------------------------ >>> *From: *"Robert Coli" >>> *To: *user@cassandra.apache.org >>> *Sent: *Tuesday, June 18, 2013 10:43:13 AM >>> *Subject: *Re: Heap is not released and streaming hangs at 0% >>> >>> On Tue, Jun 18, 2013 at 10:33 AM, srmore wrote: >>> > But then shouldn't JVM C G it eventually ? I can still see Cassandra >>> alive >>> > and kicking but looks like the heap is locked up even after the >>> traffic is >>> > long stopped. >>> >>> No, when GC system fails this hard it is often a permanent failure >>> which requires a restart of the JVM. >>> >>> > nodetool -h localhost flush didn't do much good. >>> >>> This adds support to the idea that your heap is too full, and not full >>> of memtables. >>> >>> You could try nodetool -h localhost invalidatekeycache, but that >>> probably will not free enough memory to help you. >>> >>> =Rob >>> >>> >>> >> > --20cf3074b33afb27ce04dfb3f2a7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I will take a heap dump and see whats in there rather than= guessing.=A0


On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot <<= a href=3D"mailto:btalbot@aeriagames.com" target=3D"_blank">btalbot@aeriagam= es.com> wrote:
bloom_filter_fp_chance =3D = 0.7 is probably way too large to be effective and you'll probably have = issues compacting deleted rows and get poor read performance with a value t= hat high. =A0I'd guess that anything larger than 0.1 might as well be 1= .0.

-Bryan



On Fri, Jun 21, 2013 at 5:58 AM, srmore <= ;comomore@gmail.com= > wrote:
=
On Fri, Jun 21, 2013 at 2:53 AM, aaron = morton <aaron@thelastpickle.com> wrote:
> n= odetool -h localhost flush didn't do much good.
Do you have 100's of millions of rows ?
If so see recent = discussions about reducing the bloom_filter_fp_chance and index_sampling.= =A0
Yes, I have 100's of millions of rows.
=A0

If this is an old schema you may be using the very old = setting of 0.000744 which creates a lot of bloom filters.=A0

=
bloom_filter_fp_chance value that was c= hanged from default to 0.1, looked at the filters and they are about 2.5G o= n disk and I have around 8G of heap.
I will try increasing the value to 0.7 and report my results.
It also appears to be a case of hard GC failure (as Rob me= ntioned) as the heap is never released, even after 24+ hours of idle time, = the JVM needs to be restarted to reclaim the heap.

Cheers
=A0
-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 20/06/2013, at 6:36 AM, Wei Zhu <wz1975@yahoo.com> wrote:

If you= want, you can try to force the GC through Jconsole. Memory->Perform GC.=

It theoretically triggers a full GC and when it will happen depends= on the JVM

-Wei

= From:=A0"Robert Coli" <rcoli@eventbrite.com>
To:=A0user@cassandra.apache.org
Sent:=A0= Tuesday, June 18, 2013 10:43:13 AM
Subject:=A0Re: He= ap is not released and streaming hangs at 0%

On Tue, Jun 18, 2013 at 10:33 AM, srmore <comomore@gmail.com> wrote:
> But= then shouldn't JVM C G it eventually ? I can still see Cassandra alive=
> and kicking but looks like the heap is locked up even after the traffi= c is
> long stopped.

No, when GC system fails this hard it is = often a permanent failure
which requires a restart of the JVM.

> nodetool -h localhost flush didn't do much good.

This adds = support to the idea that your heap is too full, and not full
of memtable= s.

You could try nodetool -h localhost invalidatekeycache, but that<= br> probably will not free enough memory to help you.

=3DRob
=




--20cf3074b33afb27ce04dfb3f2a7--