cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-11891) WriteTimeout during commit log replay due to MV lock
Date Wed, 25 May 2016 16:44:12 GMT
Tyler Hobbs created CASSANDRA-11891:
---------------------------------------

             Summary: WriteTimeout during commit log replay due to MV lock
                 Key: CASSANDRA-11891
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11891
             Project: Cassandra
          Issue Type: Bug
          Components: Lifecycle, Local Write-Read Paths
            Reporter: Tyler Hobbs
            Priority: Critical
             Fix For: 3.0.x, 3.x


During commit log replay, if there are materialized views, it's possible for contention on
the MV lock to cause a {{WriteTimeoutException}}.  This makes commit log replay fail, which
of course prevents the node from starting up.  This generally means that the operator has
to move the commitlog segments to avoid replay.

Here's a stacktrace of this happening on 3.0.5:

{noformat}
ERROR [main] 2016-05-25 15:10:31,120 CassandraDaemon.java:692 - Exception encountered during
startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.WriteTimeoutException:
Operation timed out - received only 0 responses.
	at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50) ~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372) ~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:624)
~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:511)
~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:406)
~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:153)
~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) ~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) ~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679) [apache-cassandra-3.0.5.jar:3.0.5]
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.WriteTimeoutException:
Operation timed out - received only 0 responses.
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.get(AbstractLocalAwareExecutorService.java:200)
~[apache-cassandra-3.0.5.jar:3.0.5]
	at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365) ~[apache-cassandra-3.0.5.jar:3.0.5]
	... 9 common frames omitted
	Suppressed: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.WriteTimeoutException:
Operation timed out - received only 0 responses.
		... 11 common frames omitted
	Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received
only 0 responses.
		at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:431)
		at org.apache.cassandra.db.Keyspace.lambda$apply$62(Keyspace.java:443)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
		at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
		at java.lang.Thread.run(Thread.java:745)
{noformat}

We should ignore the {{write_rpc_timeout}} setting while acquiring MV locks if we're on the
commitlog replay path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message