Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 03F2F18C34 for ; Thu, 30 Jul 2015 15:10:06 +0000 (UTC) Received: (qmail 33539 invoked by uid 500); 30 Jul 2015 15:10:05 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 33473 invoked by uid 500); 30 Jul 2015 15:10:05 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 33323 invoked by uid 99); 30 Jul 2015 15:10:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jul 2015 15:10:05 +0000 Date: Thu, 30 Jul 2015 15:10:05 +0000 (UTC) From: "Robbie Strickland (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-9938) Significant GC pauses MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robbie Strickland updated CASSANDRA-9938: ----------------------------------------- Attachment: gc_log.txt > Significant GC pauses > --------------------- > > Key: CASSANDRA-9938 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9938 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.04, Java 1.8.0u45 > Reporter: Robbie Strickland > Labels: gc > Attachments: gc_log.txt > > > We have an 18-node analytics cluster, running 2.1.7 patched with CASSANDRA-9662. On a couple of the nodes we are seeing very long GC pauses, especially in old gen. Eventually these nodes OOM: > {code} > ERROR [SharedPool-Worker-167] 2015-07-30 00:36:20,746 JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Java heap space > {code} > We use G1 with the following settings: > Max heap = 16G > New size = 1.6G > +UseTLAB > +ResizeTLAB > +PerfDisableSharedMem > -UseBiasedLocking > The nodes in question have average load profiles for the cluster, and caches are disabled on all tables. There is no obvious difference with the problematic nodes. Unfortunately we're currently getting an assertion error when trying to get a heap dump, or I would post that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)