incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drew Broadley <d...@broadley.org.nz>
Subject Spike in latency, one node keeps firing "Interval min > max" errors
Date Tue, 12 Feb 2013 04:38:28 GMT
Hi there,

I have a cluster of three nodes running Cassandra 1.2.0

I received alerts from my monitoring, and then discovered this huge spike
in cluster latency:
https://dl.dropbox.com/u/3444322/Screen%20Shot%202013-02-12%20at%205.07.49%20PM.png

Investigating what is going on, there is no load on any node, iostat shows
nothing more than idle operations, and I've restarted all nodes.

In the system.log I keep noticing this on ONE node only:
==> /var/log/cassandra/system.log <==
ERROR [ReadStage:563] 2013-02-12 04:31:30,013 CassandraDaemon.java (line
133) Exception in thread Thread[ReadStage:563,5,main]
java.lang.AssertionError: Interval min > max
 at
org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:250)
at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:72)
 at org.apache.cassandra.utils.IntervalTree.build(IntervalTree.java:81)
at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:175)
 at
org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:40)
at
org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:51)
 at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:224)
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:182)
 at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143)
 at
org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:86)
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:45)
 at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:134)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
 at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:286)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
 at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222)
 at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1134)
at org.apache.cassandra.db.Table.getRow(Table.java:348)
 at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1048)
 at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1506)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

-- 

Cheers,
Drew Broadley

*Broadley Speaking :)*

e:         drew@broadley.org.nz
p:         +64 (0)21 519 711
m:        P O Box 488, Wellington, New Zealand
w:        http://blog.drew.broadley.org.nz/
ln:        http://nz.linkedin.com/in/drewbroadley

Mime
View raw message