incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quintero <quinteros8...@gmail.com>
Subject RE: 0.8.6 - Cannot Restart Node, Commit Log Exception
Date Tue, 15 Nov 2011 16:13:53 GMT


Dan Hendry <dan.hendry.junk@gmail.com> wrote:

>After removing the offending commit log, the node was able to restart
>successfully. Sufficed to say that's not something I really wanted to do.
>
> 
>
>Dan
>
> 
>
>From: Dan Hendry [mailto:dan.hendry.junk@gmail.com] 
>Sent: November-15-11 10:44
>To: 'user@cassandra.apache.org'
>Subject: 0.8.6 - Cannot Restart Node, Commit Log Exception
>
> 
>
>Last night one of my nodes died inexplicably, with no log entries anywhere
>indicating the reason (cassandras log, /var/log/messages, etc). I tried to
>restart it but the node will not restart as it fails with the errors below
>when replaying the commit log. I should point out that the cluster is
>currently composed of 5 nodes running 0.8.6, and one running 1.0.2. The node
>the below exception occurred on is adjacent to the 1.0.2 node. (I have not
>fully upgraded the cluster since there have been a number of issues with
>1.0.2). I should also mention that the commit log for which playback
>ostensibly failed on (CommitLog-1321354914026.log) should have been rotated
>out *well* before the node crashed so I don't think it was corrupted by the
>crash.
>
> 
>
>Known issue or new bug? Presumably a new one since I don't see any bugs
>referenced in the changelogs between 0.8.6 and 1.0.3 which look related.
>
> 
>
>INFO [main] 2011-11-15 09:10:17,682 CommitLog.java (line 341) Finished
>reading /var/lib/cassandra/commitlog/CommitLog-1321354340572.log
>
>ERROR [MutationStage:5] 2011-11-15 09:10:19,753 AbstractCassandraDaemon.java
>(line 139) Fatal exception in thread Thread[MutationStage:5,5,main]
>
>java.lang.RuntimeException: java.lang.NullPointerException
>
>        at
>org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>
>        at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>
>        at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
>        at
>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>va:886)
>
>        at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>08)
>
>        at java.lang.Thread.run(Thread.java:662)
>
>Caused by: java.lang.NullPointerException
>
>        at
>org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309
>)
>
>        at
>org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>        ... 6 more
>
>INFO [main] 2011-11-15 09:10:19,763 CommitLog.java (line 341) Finished
>reading /var/lib/cassandra/commitlog/CommitLog-1321354914026.log
>
>ERROR [main] 2011-11-15 09:10:19,763 AbstractCassandraDaemon.java (line 358)
>Exception encountered during startup.
>
>java.lang.RuntimeException: java.util.concurrent.ExecutionException:
>java.lang.RuntimeException: java.lang.NullPointerException
>
>        at
>org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:576)
>
>        at
>org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:333)
>
>        at
>org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
>
>        at
>org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandra
>Daemon.java:200)
>
>        at
>org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassan
>draDaemon.java:341)
>
>        at
>org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)
>
>Caused by: java.util.concurrent.ExecutionException:
>java.lang.RuntimeException: java.lang.NullPointerException
>
>        at
>java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>
>        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>
>        at
>org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:572)
>
>        ... 5 more
>
>Caused by: java.lang.RuntimeException: java.lang.NullPointerException
>
>        at
>org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>
>        at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>
>        at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
>        at
>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>va:886)
>
>        at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>08)
>
>        at java.lang.Thread.run(Thread.java:662)
>
>Caused by: java.lang.NullPointerException
>
>        at
>org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309
>)
>
>        at
>org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>        ... 6 more
>
Mime
View raw message