Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9522718EFC for ; Mon, 14 Sep 2015 14:02:52 +0000 (UTC) Received: (qmail 15788 invoked by uid 500); 14 Sep 2015 14:02:46 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 15721 invoked by uid 500); 14 Sep 2015 14:02:46 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 15490 invoked by uid 99); 14 Sep 2015 14:02:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2015 14:02:46 +0000 Date: Mon, 14 Sep 2015 14:02:46 +0000 (UTC) From: "Philip Thompson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-10315) Cassandra nodes shutting down on COMMIT-LOG-ALLOCATOR error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10315?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-10315: ---------------------------------------- Description:=20 After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cassan= dra nodes shutting themselves down every 24-48 hours. The error reported i= s: {code} ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line 4= 20) Failed to allocate new commit log segments. Commit disk failure policy = is stop; terminating thread java.lang.AssertionError: attempted to delete non-existing file CommitLog-3= -1441961724221.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(Fil= eUtils.java:113) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(C= ommitLogSegment.java:161) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(C= ommitLogAllocator.java:228) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMa= yThrow(CommitLogAllocator.java:99) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnab= le.java:28) at java.lang.Thread.run(Unknown Source) {code} We=E2=80=99ve seen this error before, but now with DEBUG / TRACE on, we see= that this file has 18 entries starting at 5:14pm that look something like = this: {code} DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245)= Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/ch= ronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty i= s soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a2-= f5e6-36b6-9d7f-7fc4e109fb6e), {code} was: After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cassan= dra nodes shutting themselves down every 24-48 hours. The error reported i= s: {code} ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line 4= 20) Failed to allocate new commit log segments. Commit disk failure policy = is stop; terminating thread java.lang.AssertionError: attempted to delete non-existing file CommitLog-3= -1441961724221.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(Fil= eUtils.java:113) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(C= ommitLogSegment.java:161) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(C= ommitLogAllocator.java:228) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMa= yThrow(CommitLogAllocator.java:99) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnab= le.java:28) at java.lang.Thread.run(Unknown Source) {code} We=E2=80=99ve seen this error before, but now with DEBUG / TRACE on, we see= that this file has 18 entries starting at 5:14pm that look something like = this: DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245)= Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/ch= ronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty i= s soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a2-= f5e6-36b6-9d7f-7fc4e109fb6e), > Cassandra nodes shutting down on COMMIT-LOG-ALLOCATOR error > ----------------------------------------------------------- > > Key: CASSANDRA-10315 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1031= 5 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 16 GB, Cassandra 2.0.15,=20 > Reporter: David Loegering > > After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cass= andra nodes shutting themselves down every 24-48 hours. The error reported= is: > {code} > ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line= 420) Failed to allocate new commit log segments. Commit disk failure polic= y is stop; terminating thread > java.lang.AssertionError: attempted to delete non-existing file CommitLog= -3-1441961724221.log > at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(F= ileUtils.java:113) > at org.apache.cassandra.db.commitlog.CommitLogSegment.discard= (CommitLogSegment.java:161) > at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run= (CommitLogAllocator.java:228) > at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.run= MayThrow(CommitLogAllocator.java:99) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunn= able.java:28) > at java.lang.Thread.run(Unknown Source) > {code} > We=E2=80=99ve seen this error before, but now with DEBUG / TRACE on, we s= ee that this file has 18 entries starting at 5:14pm that look something lik= e this: > {code} > DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 24= 5) Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/= chronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty= is soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a= 2-f5e6-36b6-9d7f-7fc4e109fb6e), > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)