cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: 0.8.6 - Cannot Restart Node, Commit Log Exception
Date Tue, 15 Nov 2011 16:08:25 GMT
After removing the offending commit log, the node was able to restart
successfully. Sufficed to say that's not something I really wanted to do.

 

Dan

 

From: Dan Hendry [mailto:dan.hendry.junk@gmail.com] 
Sent: November-15-11 10:44
To: 'user@cassandra.apache.org'
Subject: 0.8.6 - Cannot Restart Node, Commit Log Exception

 

Last night one of my nodes died inexplicably, with no log entries anywhere
indicating the reason (cassandras log, /var/log/messages, etc). I tried to
restart it but the node will not restart as it fails with the errors below
when replaying the commit log. I should point out that the cluster is
currently composed of 5 nodes running 0.8.6, and one running 1.0.2. The node
the below exception occurred on is adjacent to the 1.0.2 node. (I have not
fully upgraded the cluster since there have been a number of issues with
1.0.2). I should also mention that the commit log for which playback
ostensibly failed on (CommitLog-1321354914026.log) should have been rotated
out *well* before the node crashed so I don't think it was corrupted by the
crash.

 

Known issue or new bug? Presumably a new one since I don't see any bugs
referenced in the changelogs between 0.8.6 and 1.0.3 which look related.

 

INFO [main] 2011-11-15 09:10:17,682 CommitLog.java (line 341) Finished
reading /var/lib/cassandra/commitlog/CommitLog-1321354340572.log

ERROR [MutationStage:5] 2011-11-15 09:10:19,753 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[MutationStage:5,5,main]

java.lang.RuntimeException: java.lang.NullPointerException

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NullPointerException

        at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309
)

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

        ... 6 more

INFO [main] 2011-11-15 09:10:19,763 CommitLog.java (line 341) Finished
reading /var/lib/cassandra/commitlog/CommitLog-1321354914026.log

ERROR [main] 2011-11-15 09:10:19,763 AbstractCassandraDaemon.java (line 358)
Exception encountered during startup.

java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.NullPointerException

        at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:576)

        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:333)

        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)

        at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandra
Daemon.java:200)

        at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassan
draDaemon.java:341)

        at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)

Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.NullPointerException

        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)

        at java.util.concurrent.FutureTask.get(FutureTask.java:83)

        at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:572)

        ... 5 more

Caused by: java.lang.RuntimeException: java.lang.NullPointerException

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NullPointerException

        at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309
)

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

        ... 6 more


Mime
View raw message