cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niko Kilpi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5761) Issue with secondary index sstable.
Date Mon, 28 Oct 2013 09:31:32 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806652#comment-13806652
] 

Niko Kilpi commented on CASSANDRA-5761:
---------------------------------------

Hello,

when running cassandra 1.2.4, we are suffering from this issue:

ERROR [ReadStage:7014] 2013-10-25 08:49:45,890 CassandraDaemon.java (line 174) Exception in
thread Thread[ReadStage:7014,5,main]
java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException:
EOF after 1759 bytes out of 20480
	at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:64)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException:
EOF after 1759 bytes out of 20480
	at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:91)
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:44)
	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:101)
	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
	at org.apache.cassandra.db.index.composites.CompositesSearcher$1.computeNext(CompositesSearcher.java:226)
	at org.apache.cassandra.db.index.composites.CompositesSearcher$1.computeNext(CompositesSearcher.java:161)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
	at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1466)
	at org.apache.cassandra.db.index.composites.CompositesSearcher.search(CompositesSearcher.java:85)
	at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:548)
	at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1454)
	at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:44)
	at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:58)
	... 4 more
Caused by: java.io.EOFException: EOF after 1759 bytes out of 20480
	at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:350)
	at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferUtil.java:382)
	at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:72)
	... 23 more

This is occurring roughly weekly with quite minimal usage.

> Issue with secondary index sstable.
> -----------------------------------
>
>                 Key: CASSANDRA-5761
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5761
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.5
>            Reporter: Andriy Yevsyukov
>            Priority: Critical
>
> With Cassandra 1.2.5 having issue very similar to [CASSANDRA-5225|https://issues.apache.org/jira/browse/CASSANDRA-5225]
but for secondary index sstable. Every query that uses this index fails in Hector with ConnectionTimeout
but cassandra log says that reason is:
> {noformat}
> ERROR [ReadStage:55803] 2013-07-15 12:11:35,392 CassandraDaemon.java (line 175) Exception
in thread Thread[ReadStage:55803,5,main]
> java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException:
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length
0 (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db,
19658 bytes remaining)
> 	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException:
invalid column name length 0 (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db,
19658 bytes remaining)
> 	at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:108)
> 	at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
> 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> 	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90)
> 	at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171)
> 	at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154)
> 	at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143)
> 	at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:86)
> 	at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:45)
> 	at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:134)
> 	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
> 	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:293)
> 	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> 	at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:140)
> 	at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:109)
> 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> 	at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1466)
> 	at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:82)
> 	at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:548)
> 	at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1454)
> 	at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:44)
> 	at org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1076)
> 	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
> 	... 3 more
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column
name length 0 (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db,
19658 bytes remaining)
> 	at org.apache.cassandra.db.ColumnSerializer$CorruptColumnException.create(ColumnSerializer.java:148)
> 	at org.apache.cassandra.db.RangeTombstone$Serializer.deserializeBody(RangeTombstone.java:279)
> 	at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:90)
> 	at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73)
> 	at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:104)
> 	... 30 more
> {noformat}
> We cleared CF but we have a lot of insertion and in few hours the issue happened again.
Are there any advices?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message