Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 256B71089B for ; Tue, 25 Mar 2014 22:33:41 +0000 (UTC) Received: (qmail 53086 invoked by uid 500); 25 Mar 2014 22:33:31 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 52946 invoked by uid 500); 25 Mar 2014 22:33:24 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 52809 invoked by uid 99); 25 Mar 2014 22:33:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2014 22:33:20 +0000 Date: Tue, 25 Mar 2014 22:33:20 +0000 (UTC) From: "Jonathan Ellis (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (CASSANDRA-6929) Corrupted Index File: read 8599 but expected 8600 chunks. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-6929. --------------------------------------- Resolution: Not A Problem Unless you can reproduce from specific Cassandra requests, this is most likely a hardware problem (ram, disk, raid controller are suspects). FTR, scrub will rebuild your index for you. > Corrupted Index File: read 8599 but expected 8600 chunks. > --------------------------------------------------------- > > Key: CASSANDRA-6929 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6929 > Project: Cassandra > Issue Type: Bug > Reporter: Donald Smith > > I have a 3 node cassandra cluster running 2.0.6 (we started at 2.0.1). It has several terabytes of data. We've been seeing exceptions in system.log like "Corrupted Index File ... read 21478 but expected 21479 chunks." > Here's a stack trace from one server: > {noformat} > INFO [CompactionExecutor:9109] 2014-03-24 06:55:28,148 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-compactions_in_progress@1299803435(0/0 serialized/live bytes, 1 ops) > INFO [FlushWriter:496] 2014-03-24 06:55:28,148 Memtable.java (line 331) Writing Memtable-compactions_in_progress@1299803435(0/0 serialized/live bytes, 1 ops) > INFO [FlushWriter:496] 2014-03-24 06:55:28,299 Memtable.java (line 371) Completed flushing /mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1395195644764, position=17842243) > INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,299 CompactionTask.java (line 115) Compacting [SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12861-Data.db'), SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12860-Data.db'), SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12858-Data.db'), SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db')] > ERROR [CompactionExecutor:9109] 2014-03-24 06:55:28,302 CassandraDaemon.java (line 196) Exception in thread Thread[CompactionExecutor:9109,1,main] > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db: read 21478 but expected 21479 chunks. > at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152) > at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) > at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64) > at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42) > at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db: read 21478 but expected 21479 chunks. > ... 16 more > Caused by: java.io.EOFException > at java.io.DataInputStream.readFully(Unknown Source) > at java.io.DataInputStream.readLong(Unknown Source) > at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146) > ... 15 more > INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,739 CompactionTask.java (line 275) Compacted 4 sstables to [/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12863,]. 571 bytes to 42 (~7% of original) in 439ms = 0.000091MB/s. 4 total partitions merged to 1. Partition merge counts were {2:2, } > {noformat} > Here's another example: > {noformat} > INFO [CompactionExecutor:9566] 2014-03-25 06:32:02,234 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-compactions_in_progress@1216289160(0/0 serialized/live bytes, 1 ops) > INFO [FlushWriter:474] 2014-03-25 06:32:02,234 Memtable.java (line 331) Writing Memtable-compactions_in_progress@1216289160(0/0 serialized/live bytes, 1 ops) > INFO [FlushWriter:474] 2014-03-25 06:32:02,445 Memtable.java (line 371) Completed flushing /mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-33701-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1395185650610, position=4461144) > ERROR [CompactionExecutor:9566] 2014-03-25 06:32:02,447 CassandraDaemon.java (line 196) Exception in thread Thread[CompactionExecutor:9566,1,main] > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db: read 8599 but expected 8600 chunks. > at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152) > at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) > at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64) > at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42) > at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db: read 8599 but expected 8600 chunks. > ... 16 more > Caused by: java.io.EOFException > at java.io.DataInputStream.readFully(Unknown Source) > at java.io.DataInputStream.readLong(Unknown Source) > at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146) > ... 15 more > {noformat} > Here are multiple examples (f is an alias for "grep -i"): > {noformat} > cass1 /var/log/cassandra> f exception system.log | f corrupted | f chunks > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db: read 18848 but expected 18849 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db: read 18848 but expected 18849 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db: read 19930 but expected 19931 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db: read 19930 but expected 19931 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db: read 19607 but expected 19608 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db: read 19607 but expected 19608 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db: read 18425 but expected 18426 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db: read 18425 but expected 18426 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db: read 18648 but expected 18649 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db: read 18648 but expected 18649 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db: read 18882 but expected 18883 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db: read 18882 but expected 18883 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db: read 18949 but expected 18950 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db: read 18949 but expected 18950 chunks. > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db: read 21478 but expected 21479 chunks. > Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db: read 21478 but expected 21479 chunks. > {noformat} > Let me know if you need more info. I wouldn't know how to reproduce this from a simple example. -- This message was sent by Atlassian JIRA (v6.2#6252)