cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-6929) Corrupted Index File: read 8599 but expected 8600 chunks.
Date Tue, 25 Mar 2014 22:33:20 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis resolved CASSANDRA-6929.
---------------------------------------

    Resolution: Not A Problem

Unless you can reproduce from specific Cassandra requests, this is most likely a hardware
problem (ram, disk, raid controller are suspects).

FTR, scrub will rebuild your index for you.

> Corrupted Index File: read 8599 but expected 8600 chunks.
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-6929
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6929
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Donald Smith
>
> I have a 3 node cassandra cluster running 2.0.6 (we started at 2.0.1). It has several
terabytes of data. We've been seeing exceptions in system.log like "Corrupted Index File ...
read 21478 but expected 21479 chunks."
> Here's a stack trace from one server:
> {noformat}
>  INFO [CompactionExecutor:9109] 2014-03-24 06:55:28,148 ColumnFamilyStore.java (line
785) Enqueuing flush of Memtable-compactions_in_progress@1299803435(0/0 serialized/live bytes,
1 ops)
>  INFO [FlushWriter:496] 2014-03-24 06:55:28,148 Memtable.java (line 331) Writing Memtable-compactions_in_progress@1299803435(0/0
serialized/live bytes, 1 ops)
>  INFO [FlushWriter:496] 2014-03-24 06:55:28,299 Memtable.java (line 371) Completed flushing
/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1395195644764, position=17842243)
>  INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,299 CompactionTask.java (line 115)
Compacting [SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12861-Data.db'),
SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12860-Data.db'),
SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12858-Data.db'),
SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db')]
> ERROR [CompactionExecutor:9109] 2014-03-24 06:55:28,302 CassandraDaemon.java (line 196)
Exception in thread Thread[CompactionExecutor:9109,1,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
>         at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152)
>         at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
>         at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
>         at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
>         at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
>         ... 16 more
> Caused by: java.io.EOFException
>         at java.io.DataInputStream.readFully(Unknown Source)
>         at java.io.DataInputStream.readLong(Unknown Source)
>         at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146)
>         ... 15 more
>  INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,739 CompactionTask.java (line 275)
Compacted 4 sstables to [/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12863,].
 571 bytes to 42 (~7% of original) in 439ms = 0.000091MB/s.  4 total partitions merged to
1.  Partition merge counts were {2:2, }
> {noformat}
> Here's another example:
> {noformat}
>  INFO [CompactionExecutor:9566] 2014-03-25 06:32:02,234 ColumnFamilyStore.java (line
785) Enqueuing flush of Memtable-compactions_in_progress@1216289160(0/0 serialized/live bytes,
1 ops)
>  INFO [FlushWriter:474] 2014-03-25 06:32:02,234 Memtable.java (line 331) Writing Memtable-compactions_in_progress@1216289160(0/0
serialized/live bytes, 1 ops)
>  INFO [FlushWriter:474] 2014-03-25 06:32:02,445 Memtable.java (line 371) Completed flushing
/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-33701-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1395185650610, position=4461144)
> ERROR [CompactionExecutor:9566] 2014-03-25 06:32:02,447 CassandraDaemon.java (line 196)
Exception in thread Thread[CompactionExecutor:9566,1,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db:
read 8599 but expected 8600 chunks.
>         at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152)
>         at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
>         at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
>         at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
>         at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db:
read 8599 but expected 8600 chunks.
>         ... 16 more
> Caused by: java.io.EOFException
>         at java.io.DataInputStream.readFully(Unknown Source)
>         at java.io.DataInputStream.readLong(Unknown Source)
>         at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146)
>         ... 15 more
> {noformat}
> Here are multiple examples (f is an alias for "grep -i"):
> {noformat}
> cass1 /var/log/cassandra> f exception system.log  | f corrupted | f chunks
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db:
read 18848 but expected 18849 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db:
read 18848 but expected 18849 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db:
read 19930 but expected 19931 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db:
read 19930 but expected 19931 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db:
read 19607 but expected 19608 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db:
read 19607 but expected 19608 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db:
read 18425 but expected 18426 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db:
read 18425 but expected 18426 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db:
read 18648 but expected 18649 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db:
read 18648 but expected 18649 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db:
read 18882 but expected 18883 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db:
read 18882 but expected 18883 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db:
read 18949 but expected 18950 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db:
read 18949 but expected 18950 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: Corrupted
Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
> Caused by: java.io.IOException: Corrupted Index File /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
read 21478 but expected 21479 chunks.
> {noformat}
> Let me know if you need more info. I wouldn't know how to reproduce this from a simple
example.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message