Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 29 Jun 2015 17:33:04 +0000 (UTC)
From: "mlowicki (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12841392.1435599164000.46006.1435599184857@Atlassian.JIRA>
In-Reply-To: <JIRA.12841392.1435599164000@Atlassian.JIRA>
References: <JIRA.12841392.1435599164000@Atlassian.JIRA>
 <JIRA.12841392.1435599164029@arcas>
Subject: [jira] [Created] (CASSANDRA-9681) Memtable heap size grows and many
 long GC pauses are triggered
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

mlowicki created CASSANDRA-9681:
-----------------------------------

             Summary: Memtable heap size grows and many long GC pauses are triggered
                 Key: CASSANDRA-9681
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9681
             Project: Cassandra
          Issue Type: Bug
         Environment: C* 2.1.7, Debian Wheezy
            Reporter: mlowicki
            Priority: Critical


C* 2.1.7 cluster is behaving really bad after 1-2 days. {{gauges.cassandra.jmx.org.apache.cassandra.metrics.ColumnFamily.AllMemtablesHeapSize.Value}} jumps to 7 GB (https://www.dropbox.com/s/vraggy292erkzd2/Screenshot%202015-06-29%2019.12.53.png?dl=0) on 3/6 nodes in each data center and then there are many long GC pauses. Cluster is using default heap size values ({{-Xms8192M -Xmx8192M -Xmn2048M}})

Before C* 2.1.5 memtables heap size was basically constant ~500MB (https://www.dropbox.com/s/fjdywik5lojstvn/Screenshot%202015-06-29%2019.30.00.png?dl=0)

After restarting all nodes is behaves stable for 1-2days. Today I've done that and long GC pauses are gone (~18:00 https://www.dropbox.com/s/7vo3ynz505rsfq3/Screenshot%202015-06-29%2019.28.37.png?dl=0). The only pattern we've found so far is that long GC  pauses are happening basically at the same time on all nodes in the same data center - even on the ones where memtables heap size is not growing.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)