Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B2C7B10E2F for ; Tue, 20 Oct 2015 13:21:34 +0000 (UTC) Received: (qmail 93804 invoked by uid 500); 20 Oct 2015 13:21:28 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 93770 invoked by uid 500); 20 Oct 2015 13:21:28 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 93684 invoked by uid 99); 20 Oct 2015 13:21:28 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2015 13:21:28 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id DC1E72C1F6A for ; Tue, 20 Oct 2015 13:21:27 +0000 (UTC) Date: Tue, 20 Oct 2015 13:21:27 +0000 (UTC) From: "Sharvanath Pathak (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (CASSANDRA-10479) Handling partially written sstables on node crashes MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharvanath Pathak resolved CASSANDRA-10479. ------------------------------------------- Resolution: Invalid On more investigation it seems that the real issue was https://issues.apache.org/jira/browse/CASSANDRA-10534. > Handling partially written sstables on node crashes > --------------------------------------------------- > > Key: CASSANDRA-10479 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10479 > Project: Cassandra > Issue Type: Bug > Reporter: Sharvanath Pathak > > Currently a power loss can potentially require manual intervention to bring Cassandra back up. Essentially, these partially written SStables are considered as corrupt, and we see the following trace quite often on hard reboots: > {noformat} > INFO [SSTableBatchOpen:1] 2015-09-29 19:24:39,170 SSTableReader.java:478 - Opening /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-13368 (79 bytes) > ERROR [SSTableBatchOpen:1] 2015-09-29 19:24:39,177 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException > at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) ~[apache-cassandra-2.1.9.jar:2.1.9] > at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-2.1.9.jar:2.1.9] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80] > Caused by: java.io.EOFException: null > at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.7.0_80] > at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.7.0_80] > at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.7.0_80] > at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) ~[apache-cassandra-2.1.9.jar:2.1.9] > ... 14 common frames omitted > {noformat} > Deleting partially written SStables might be a perfectly valid thing to do (given that the data is present in commitlogs). -- This message was sent by Atlassian JIRA (v6.3.4#6332)