incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Rhoden <jacob.rho...@me.com>
Subject Recovering from mystical cassandra errors.
Date Tue, 29 Oct 2013 23:40:29 GMT
Hi Guys,

Im testing out Cassandra with a 2.0.1 install, just like with 1.2.8 I last tested, from time
to time I am coming across errors that prevent Cassandra starting up.

For example: Yesterday I installed this instance, created and removed some “tables” using
cql3 and inserted/updated/truncated some tables. Overnight, not sure why, cassandra stopped
running and won’t start up again:

 INFO 10:34:37,776 Finished reading /cassandra/commitlog/CommitLog-3-1382935041799.log
 INFO 10:34:37,776 Replaying /cassandra/commitlog/CommitLog-3-1383089534960.log
 INFO 10:34:37,796 Finished reading /cassandra/commitlog/CommitLog-3-1383089534960.log
 INFO 10:34:37,796 Skipped 11 mutations from unknown (probably removed) CF with id 941b25fd-703f-3013-aadd-40b18cbcd9cd
ERROR 10:34:37,797 Exception encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
index (2) must be less than size (2)
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
	at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400)
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:105)
	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:147)
	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: index
(2) must be less than size (2)
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407)
	... 7 more

Normally I just delete the files and restart the node. If you are in a production environment,
how do you deal with this type of issue?

(I suspect there may be some sort of bug with replaying commit logs in 2.0.1, probably associated
with replaying a commit log after you have deleted a table, and recreated with a different
definition.)

Best regards,
Jacob
Mime
View raw message