incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Wirt" <chris.w...@struq.com>
Subject 2.0.1 counter replicate on write error
Date Fri, 27 Sep 2013 10:50:15 GMT
Hello,

 

I've started to see a slightly worrying error appear in our logs
occasionally. We're writing at 400qps per machine and I only see this appear
every 5-10minutes.

 

Seems to have started when I switched us to using the hsha thrift server
this morning. We've been running 2.0.1 ran off the sync thrift server since
yesterday without seeing this error.  But might not be related.

 

There are some machines in another DC still running 1.2.10.

 

Anyone seen this before? Have any insight?

 

ERROR [ReplicateOnWriteStage:19] 2013-09-27 10:17:14,778
CassandraDaemon.java (line 185) Exception in thread
Thread[ReplicateOnWriteStage:19,5,main]

java.lang.AssertionError: DecoratedKey(-1754949563326053382,
a414b0c07f0547f8a75410555716ced6) != DecoratedKey(-1754949563326053382,
aeadcec8184445d4ab631ef4250927d0) in
/disk3/cassandra/data/struqrealtime/counters/struqrealtime-counters-jb-83195
3-Data.db

        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableName
sIterator.java:114)

        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNa
mesIterator.java:62)

        at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(Nam
esQueryFilter.java:87)

        at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFil
ter.java:62)

        at
org.apache.cassandra.db.CollationController.collectAllData(CollationControll
er.java:249)

        at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont
roller.java:53)

        at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto
re.java:1468)

        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1294)

        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:332)

        at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadComma
nd.java:55)

        at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutat
ion.java:100)

        at
org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:
1107)

        at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy
.java:1897)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:724)


Mime
View raw message