cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Donald Smith (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-6929) Corrupted Index File: read 8599 but expected 8600 chunks.
Date Tue, 25 Mar 2014 22:09:14 GMT
Donald Smith created CASSANDRA-6929:
---------------------------------------

             Summary: Corrupted Index File: read 8599 but expected 8600 chunks.
                 Key: CASSANDRA-6929
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6929
             Project: Cassandra
          Issue Type: Bug
            Reporter: Donald Smith


I have a 3 node cassandra cluster running 2.0.6 (we started at 2.0.1). It has several terabytes
of data. We've been seeing exceptions in system.log like "Corrupted Index File ... read 21478
but expected 21479 chunks."



Here's a stack trace from one server:
{noformat}
 INFO [CompactionExecutor:9109] 2014-03-24 06:55:28,148 ColumnFamilyStore.java (line 785)
Enqueuing flush of Memtable-compactions_in_progress@1299803435(0/0 serialized/live bytes,
1 ops)
 INFO [FlushWriter:496] 2014-03-24 06:55:28,148 Memtable.java (line 331) Writing Memtable-compactions_in_progress@1299803435(0/0
serialized/live bytes, 1 ops)
 INFO [FlushWriter:496] 2014-03-24 06:55:28,299 Memtable.java (line 371) Completed flushing
/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1395195644764, position=17842243)
 INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,299 CompactionTask.java (line 115) Compacting
[SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12861-Data.db'),
SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12860-Data.db'),
SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12858-Data.db'),
SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db')]
ERROR [CompactionExecutor:9109] 2014-03-24 06:55:28,302 CassandraDaemon.java (line 196) Exception
in thread Thread[CompactionExecutor:9109,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
        at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152)
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
        at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
        at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
        ... 16 more
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(Unknown Source)
        at java.io.DataInputStream.readLong(Unknown Source)
        at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146)
        ... 15 more
 INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,739 CompactionTask.java (line 275) Compacted
4 sstables to [/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12863,].
 571 bytes to 42 (~7% of original) in 439ms = 0.000091MB/s.  4 total partitions merged to
1.  Partition merge counts were {2:2, }
{noformat}
Here's another example:



{noformat}
 INFO [CompactionExecutor:9566] 2014-03-25 06:32:02,234 ColumnFamilyStore.java (line 785)
Enqueuing flush of Memtable-compactions_in_progress@1216289160(0/0 serialized/live bytes,
1 ops)
 INFO [FlushWriter:474] 2014-03-25 06:32:02,234 Memtable.java (line 331) Writing Memtable-compactions_in_progress@1216289160(0/0
serialized/live bytes, 1 ops)
 INFO [FlushWriter:474] 2014-03-25 06:32:02,445 Memtable.java (line 371) Completed flushing
/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-33701-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1395185650610, position=4461144)
ERROR [CompactionExecutor:9566] 2014-03-25 06:32:02,447 CassandraDaemon.java (line 196) Exception
in thread Thread[CompactionExecutor:9566,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db:
read 8599 but expected 8600 chunks.
        at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152)
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
        at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
        at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db:
read 8599 but expected 8600 chunks.
        ... 16 more
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(Unknown Source)
        at java.io.DataInputStream.readLong(Unknown Source)
        at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146)
        ... 15 more
{noformat}

Here are multiple examples (f is an alias for "grep -i"):
{noformat}
cass1 /var/log/cassandra> f exception system.log  | f corrupted | f chunks
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db:
read 18848 but expected 18849 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db:
read 18848 but expected 18849 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db:
read 19930 but expected 19931 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db:
read 19930 but expected 19931 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db:
read 19607 but expected 19608 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db:
read 19607 but expected 19608 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db:
read 18425 but expected 18426 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db:
read 18425 but expected 18426 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db:
read 18648 but expected 18649 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db:
read 18648 but expected 18649 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db:
read 18882 but expected 18883 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db:
read 18882 but expected 18883 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db:
read 18949 but expected 18950 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db:
read 18949 but expected 18950 chunks.
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted Index
File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
{noformat}

Let me know is you need more info. I wouldn't know how to reproduce this from a simple example.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message