Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 593A010C4B for ; Wed, 19 Jun 2013 18:37:08 +0000 (UTC) Received: (qmail 58069 invoked by uid 500); 19 Jun 2013 18:37:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 58053 invoked by uid 500); 19 Jun 2013 18:37:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 58045 invoked by uid 99); 19 Jun 2013 18:37:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jun 2013 18:37:05 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLYTO_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [216.109.115.141] (HELO nm47-vm6.bullet.mail.bf1.yahoo.com) (216.109.115.141) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jun 2013 18:36:59 +0000 Received: from [98.139.214.32] by nm47.bullet.mail.bf1.yahoo.com with NNFMP; 19 Jun 2013 18:36:38 -0000 Received: from [98.139.212.202] by tm15.bullet.mail.bf1.yahoo.com with NNFMP; 19 Jun 2013 18:36:38 -0000 Received: from [127.0.0.1] by omp1011.mail.bf1.yahoo.com with NNFMP; 19 Jun 2013 18:36:38 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 680156.88067.bm@omp1011.mail.bf1.yahoo.com Received: (qmail 57095 invoked by uid 60001); 19 Jun 2013 18:36:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1371666998; bh=fMF0slDqnEH5EJtt43mveRa1frlKBQPDx0Pp+31pyC8=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=crDRyHGbRqmEw9siAIgAeftR+QWFtWzrF5yYW6tHcu4aGjGQXoSohujU3dDBShnd5zPrCyoFw60sIzUME3+U43FKdcMJXwBbPitiDWY6yMGEHl4/rUsUvE1eu9VwEuMOylMQjWOSLFDnfb4QFW318oLJMX6nautihexRdOWXOUk= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=h41VYNr7fGqHYwwqwxfgURXYW8HqkGCvplal1FvNPerIKrTGVsYySd/rXaAQNwCCEViFNLxFKCdObVDs4fM3s5fR26VaHH+2OYS+xdPXTg7gNm4ss1s9iLBQS/gG/HEC8ST7Uu2XNgl0cNblGC8fMLTNWnortYzGPCT3R06YZVw=; X-YMail-OSG: tbMLKLgVM1lAcCP_FCiXsoCrZ4VYWgX2HLdIs0_CWKtAKvG yIAb5vQ4fHw65g8Aj_n0Acos_dh0OzUOXlL5hDcvXXG5J_rlEYVfhxRPAllq Bfku_SdHTgqbTwLopmKGjkd2sw0k0JJOQtUoJl6om_7f3uTZQ4eLGuLP_s4w mDNvCh203CCppaEBgY.SPQ2ZQaC5k4BWHBh66oCQ._FPtoHKyDfdilXPPeWs XyS6rvAx5PduTlioOFf4AZtG35iRVZL4znp6e6P6JdxkkBaoJ.SUMhGeJOub hm165ruK8SedoPymDe36EElCTSUvYLE3P11..Yq7YmzV7X455fGXQF7UfwUn 1HPBpZPZ6EaLzAlfz7yJPd2uIioGkZlnuAxYv7rHZx01zqV6Rvok5p6EdKoq rxXL4WfITq6gBvMe9t.Ij0ZCE.2dUeAxV77Sp9ggcMipx1NHWDSS5wVziDFy C_Tcmh7JSUvxLdNZz.1.9Nt.1QK3s2EyzB23CvJ8EHGdcuLX9rzYgbJqOLtU ZQT.1rzyJiTKlXgm91TxVEyol94L2N82.q8NKrw-- Received: from [208.185.20.30] by web160905.mail.bf1.yahoo.com via HTTP; Wed, 19 Jun 2013 11:36:38 PDT X-Mailer: YahooMailWebService/0.8.147.553 Message-ID: <1371666998.53260.GenericBBA@web160905.mail.bf1.yahoo.com> Date: Wed, 19 Jun 2013 11:36:38 -0700 (PDT) From: Wei Zhu Reply-To: Wei Zhu Subject: Re: Heap is not released and streaming hangs at 0% To: user@cassandra.apache.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-1879531-375958801-1371666998=:53260" X-Virus-Checked: Checked by ClamAV on apache.org ---1879531-375958801-1371666998=:53260 Content-Type: text/plain; charset=us-ascii If you want, you can try to force the GC through Jconsole. Memory->Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei ----- Original Message ----- From: "Robert Coli" To: user@cassandra.apache.org Sent: Tuesday, June 18, 2013 10:43:13 AM Subject: Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore wrote: > But then shouldn't JVM C G it eventually ? I can still see Cassandra alive > and kicking but looks like the heap is locked up even after the traffic is > long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. > nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob ---1879531-375958801-1371666998=:53260 Content-Type: text/html; charset=us-ascii
If you want, you can try to force the GC through Jconsole. Memory->Perform GC.

It theoretically triggers a full GC and when it will happen depends on the JVM

-Wei


From: "Robert Coli" <rcoli@eventbrite.com>
To: user@cassandra.apache.org
Sent: Tuesday, June 18, 2013 10:43:13 AM
Subject: Re: Heap is not released and streaming hangs at 0%

On Tue, Jun 18, 2013 at 10:33 AM, srmore <comomore@gmail.com> wrote:
> But then shouldn't JVM C G it eventually ? I can still see Cassandra alive
> and kicking but looks like the heap is locked up even after the traffic is
> long stopped.

No, when GC system fails this hard it is often a permanent failure
which requires a restart of the JVM.

> nodetool -h localhost flush didn't do much good.

This adds support to the idea that your heap is too full, and not full
of memtables.

You could try nodetool -h localhost invalidatekeycache, but that
probably will not free enough memory to help you.

=Rob

---1879531-375958801-1371666998=:53260--