Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B84FD177CC for ; Fri, 16 Oct 2015 14:18:09 +0000 (UTC) Received: (qmail 36384 invoked by uid 500); 16 Oct 2015 14:18:06 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 36357 invoked by uid 500); 16 Oct 2015 14:18:06 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 36341 invoked by uid 99); 16 Oct 2015 14:18:06 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2015 14:18:06 +0000 Date: Fri, 16 Oct 2015 14:18:06 +0000 (UTC) From: "Marcus Eriksson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-10515) Commit logs back up with move to 2.1.10 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960767#comment-14960767 ] Marcus Eriksson commented on CASSANDRA-10515: --------------------------------------------- if the 5k sstables are all in L0, then yes, that can be painful > Commit logs back up with move to 2.1.10 > --------------------------------------- > > Key: CASSANDRA-10515 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10515 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: redhat 6.5, cassandra 2.1.10 > Reporter: Jeff Griffith > Assignee: Branimir Lambov > Priority: Critical > Labels: commitlog, triage > Attachments: C5commitLogIncrease.jpg, CommitLogProblem.jpg, CommitLogSize.jpg, RUN3tpstats.jpg, cassandra.yaml, cfstats-clean.txt, stacktrace.txt, system.log.clean > > > After upgrading from cassandra 2.0.x to 2.1.10, we began seeing problems where some nodes break the 12G commit log max we configured and go as high as 65G or more before it restarts. Once it reaches the state of more than 12G commit log files, "nodetool compactionstats" hangs. Eventually C* restarts without errors (not sure yet whether it is crashing but I'm checking into it) and the cleanup occurs and the commit logs shrink back down again. Here is the nodetool compactionstats immediately after restart. > {code} > jgriffith@prod1xc1.c2.bf1:~$ ndc > pending tasks: 2185 > compaction type keyspace table completed total unit progress > Compaction SyncCore *cf1* 61251208033 170643574558 bytes 35.89% > Compaction SyncCore *cf2* 19262483904 19266079916 bytes 99.98% > Compaction SyncCore *cf3* 6592197093 6592316682 bytes 100.00% > Compaction SyncCore *cf4* 3411039555 3411039557 bytes 100.00% > Compaction SyncCore *cf5* 2879241009 2879487621 bytes 99.99% > Compaction SyncCore *cf6* 21252493623 21252635196 bytes 100.00% > Compaction SyncCore *cf7* 81009853587 81009854438 bytes 100.00% > Compaction SyncCore *cf8* 3005734580 3005768582 bytes 100.00% > Active compaction remaining time : n/a > {code} > I was also doing periodic "nodetool tpstats" which were working but not being logged in system.log on the StatusLogger thread until after the compaction started working again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)