Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9D38710DBF for ; Wed, 27 Nov 2013 10:01:41 +0000 (UTC) Received: (qmail 71110 invoked by uid 500); 27 Nov 2013 10:01:41 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 70611 invoked by uid 500); 27 Nov 2013 10:01:40 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 70449 invoked by uid 99); 27 Nov 2013 10:01:40 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Nov 2013 10:01:40 +0000 Date: Wed, 27 Nov 2013 10:01:40 +0000 (UTC) From: "Andriy Yevsyukov (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-5761) Issue with secondary index sstable. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833624#comment-13833624 ] Andriy Yevsyukov commented on CASSANDRA-5761: --------------------------------------------- Hi, Not sure whether it fixes the problem, but that's what we found. 1. Problem happens when often reset colum families- we are in active development stage so very often reset colum families(drop, create, load test data) 2. invalidate cache doesn't help. Our solution: We dropped keyspaces and deleted ALL files in data folder We stopped to reset column families For now everything is fine > Issue with secondary index sstable. > ----------------------------------- > > Key: CASSANDRA-5761 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5761 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.2.5 > Reporter: Andriy Yevsyukov > Priority: Critical > > With Cassandra 1.2.5 having issue very similar to [CASSANDRA-5225|https://issues.apache.org/jira/browse/CASSANDRA-5225] but for secondary index sstable. Every query that uses this index fails in Hector with ConnectionTimeout but cassandra log says that reason is: > {noformat} > ERROR [ReadStage:55803] 2013-07-15 12:11:35,392 CassandraDaemon.java (line 175) Exception in thread Thread[ReadStage:55803,5,main] > java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0 (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db, 19658 bytes remaining) > at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0 (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db, 19658 bytes remaining) > at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:108) > at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39) > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90) > at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171) > at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154) > at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143) > at org.apache.cassandra.utils.MergeIterator$ManyToOne.(MergeIterator.java:86) > at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:45) > at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:134) > at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84) > at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:293) > at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) > at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) > at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) > at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:140) > at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:109) > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1466) > at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:82) > at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:548) > at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1454) > at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:44) > at org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1076) > at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578) > ... 3 more > Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid column name length 0 (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db, 19658 bytes remaining) > at org.apache.cassandra.db.ColumnSerializer$CorruptColumnException.create(ColumnSerializer.java:148) > at org.apache.cassandra.db.RangeTombstone$Serializer.deserializeBody(RangeTombstone.java:279) > at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:90) > at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73) > at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:104) > ... 30 more > {noformat} > We cleared CF but we have a lot of insertion and in few hours the issue happened again. Are there any advices? -- This message was sent by Atlassian JIRA (v6.1#6144)