Update: I've reverted hinted_handoff_enabled back to its default value of true and the errors stopped. Is this just a coincidence, or could be related? 



On Sun, Oct 6, 2013 at 7:23 PM, Ran Tavory <rantav@gmail.com> wrote:
Pardon me, now with the appropriate subject line...

Hi, I have a small cluster of 1.2.6 and after some config changes I started seeing errors int the logs.

Not sure that's related, but the changes I performed were to disable hinted handoff and disable auto snapshot. I'll try to reverte these, see if the picture changes. 

But anyway, that seems like a bug, right?

I see this across many nodes, not only one. 

ERROR [ReplicateOnWriteStage:105] 2013-10-06 16:13:13,799 CassandraDaemon.java (line 192) Exception in thread Thread[ReplicateOnWriteStage:105,5,main]
java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) != DecoratedKey(-1854619418400985942, 000338393900000a4769676f707469782d3100) in /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [ReplicateOnWriteStage:82] 2013-10-06 16:13:14,249 CassandraDaemon.java (line 192) Exception in thread Thread[ReplicateOnWriteStage:82,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek to position 2171332 in /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db (1250125 bytes) in read-only mode
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: unable to seek to position 2171332 in /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db (1250125 bytes) in read-only mode
at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1054)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)

--



--
/Ran
http://tavory.com