cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Danh Kieu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-5890) Exception: CorruptSSTableException after nightly 'enqueuing flush'
Date Wed, 14 Aug 2013 18:14:48 GMT
Danh Kieu created CASSANDRA-5890:
------------------------------------

             Summary: Exception: CorruptSSTableException after nightly 'enqueuing flush'
                 Key: CASSANDRA-5890
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5890
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Release: 1.2.5
2 Cassandra nodes in a cluster
OS: Ubuntu 12.04.2 LTS
CQL: 3.0.2
            Reporter: Danh Kieu


The 2 Cassandra nodes have worked well for 2 days. However, after a nightly 'enqueuing flush'
process, a few column families (in all keyspaces) are corrupted and the following error is
observed in log file while trying to access the CF:

ERROR [ReadStage:413277] 2013-08-14 10:16:13,322 CassandraDaemon.java (line 175) Exception
in thread Thread[ReadStage:413277,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:65)
        at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
        at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
        at org.apache.cassandra.db.Table.getRow(Table.java:347)
        at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
        at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:380)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:116)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
        ... 14 more
ERROR [ReadStage:413292] 2013-08-14 10:16:53,700 CassandraDaemon.java (line 175) Exception
in thread Thread[ReadStage:413292,5,main]
java.lang.AssertionError: DecoratedKey(-8619398030348476976, 796177616431406d6e75626f2e636f6d)
!= DecoratedKey(-8430385117828588592, 79624079622e636f6d) in /var/lib/cassandra/data/newui/user/newui-user-ic-1-Data.db
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
        at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
        at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
        at org.apache.cassandra.db.Table.getRow(Table.java:347)
        at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)


Here the traces of the 'enqueuing flush' process:
 INFO [OptionalTasks:1] 2013-08-14 04:01:50,586 ColumnFamilyStore.java (line 631) Enqueuing
flush of Memtable-user@848766585(3130/3130 serialized/live bytes, 96 ops)
 INFO [FlushWriter:381] 2013-08-14 04:01:50,632 Memtable.java (line 461) Writing Memtable-user@848766585(3130/3130
serialized/live bytes, 96 ops)
 INFO [FlushWriter:381] 2013-08-14 04:01:50,641 Memtable.java (line 495) Completed flushing
/var/lib/cassandra/data/newui/user/newui-user-ic-1-Data.db (1513 bytes) for commitlog position
ReplayPosition(segmentId=1373063890581, position=9254)

The issue had happened quite regularly and on a random CF. Please note that we don't have
a lot of data (max 20 records per CF).

We've to run 'nodetool scrub' and 'nodetool repair' to get out of troubles. Please let me
know if this issue is related to a configuration, a bad use of Cassandra or a real problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message