Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7C306200BAE for ; Fri, 28 Oct 2016 09:54:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 7ADBD160AF5; Fri, 28 Oct 2016 07:54:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C1A31160ADD for ; Fri, 28 Oct 2016 09:54:04 +0200 (CEST) Received: (qmail 17197 invoked by uid 500); 28 Oct 2016 07:53:59 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 17021 invoked by uid 99); 28 Oct 2016 07:53:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Oct 2016 07:53:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id B56062C0D55 for ; Fri, 28 Oct 2016 07:53:58 +0000 (UTC) Date: Fri, 28 Oct 2016 07:53:58 +0000 (UTC) From: "Arvind Nithrakashyap (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 28 Oct 2016 07:54:05 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15614667#comment-15614667 ] Arvind Nithrakashyap commented on CASSANDRA-12539: -------------------------------------------------- It looks like a crash while writing the logfile can lead to a zero byte file. The following patch which causes a crash at a specific point reliably produces a commit log full of zeros {noformat} diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java index 0a03c3c..20cddf8 100644 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java @@ -153,6 +153,7 @@ public abstract class CommitLogSegment descriptor = new CommitLogDescriptor(id, commitLog.configuration.getCompressorClass()); logFile = new File(commitLog.location, descriptor.fileName()); + logger.error("location="+descriptor.fileName()); try { channel = FileChannel.open(logFile.toPath(), StandardOpenOption.WRITE, StandardOpenOption.READ, StandardOpenOption.CREATE); @@ -164,6 +165,9 @@ public abstract class CommitLogSegment } buffer = createBuffer(commitLog); + if (true) { + throw new IllegalArgumentException("Here!"); + } // write the header CommitLogDescriptor.writeHeader(buffer, descriptor); endOfBuffer = buffer.capacity(); {noformat} > Empty CommitLog prevents restart > -------------------------------- > > Key: CASSANDRA-12539 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12539 > Project: Cassandra > Issue Type: Bug > Reporter: Stefano Ortolani > > A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike other time) restarting simply fails. > Checking the logs showed: > {noformat} > ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization. > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file /data/cassandra/commitlog/CommitLog-6-1468235564433.log > at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557) [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685) [apache-cassandra-3.0.8.jar:3.0.8] > INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 - Configuration location: file:/etc/cassandra/cassandra.yaml > {noformat} > Deleting the empty file fixes the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)