cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang, Charles" <Cong.Zh...@fmr.com>
Subject Timeout while waiting for workers when flushing pool
Date Wed, 01 Jun 2016 14:55:06 GMT
We have a 4 nodes, two dc's test cluster. All of them have datastax enterprise installed and
running. One dc is the Cassandra dc, and the other is the Solr dc. We first used sstableloader
to stream 1 billion rows into the cluster. After that was done, we created a Solr core using
resource auto-generation. It started indexing fine,  for a while, and then one of the solr
nodes went down. And the system log shows this:

ERROR [NonPeriodicTasks:1] 2016-05-31 17:47:36,560  CassandraDaemon.java:229 - Exception in
thread Thread[NonPeriodicTasks:1,5,main]
java.lang.RuntimeException: Timeout while waiting for workers when flushing pool zootopia.ltb
Index; current timeout is 300000 millis, consider increasing it, or reducing load on the node.
Failure to flush may cause excessive growth of Cassandra commit log.

        at com.datastax.bdp.search.solr.core.CassandraCoreContainer.doShutdown(CassandraCoreContainer.java:1081)
~[dse-search-4.8.7.jar:4.8.7]
        at com.datastax.bdp.search.solr.core.CassandraCoreContainer.access$100(CassandraCoreContainer.java:99)
~[dse-search-4.8.7.jar:4.8.7]
        at com.datastax.bdp.search.solr.core.CassandraCoreContainer$1.run(CassandraCoreContainer.java:626)
~[dse-search-4.8.7.jar:4.8.7]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
~[na:1.8.0_91]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
~[na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for workers when flushing
pool zootopia.ltb Index; current timeout is 300000 millis, consider increasing it, or reducing
load on the node.
Failure to flush may cause excessive growth of Cassandra commit log.

        at com.datastax.bdp.concurrent.WorkPool.doFlushError(WorkPool.java:598) ~[dse-core-4.8.7.jar:4.8.7]
        at com.datastax.bdp.concurrent.WorkPool.doTwoPhaseFlush(WorkPool.java:559) ~[dse-core-4.8.7.jar:4.8.7]
        at com.datastax.bdp.concurrent.WorkPool.doFlush(WorkPool.java:523) ~[dse-core-4.8.7.jar:4.8.7]
        at com.datastax.bdp.concurrent.WorkPool.shutdown(WorkPool.java:461) ~[dse-core-4.8.7.jar:4.8.7]
        at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.shutdownIndexUpdates(AbstractSolrSecondaryIndex.java:534)
~[dse-search-4.8.7.jar:4.8.7]
        at com.datastax.bdp.search.solr.core.CassandraCoreContainer.doShutdown(CassandraCoreContainer.java:1076)
~[dse-search-4.8.7.jar:4.8.7]
        ... 9 common frames omitted


When I restarted the solr nodes, the num of docs showed a very small number, much smaller
than the last number we saw before it went down...

I saw this was a known issue in datastax 4.8 and was resolved in 4.8.1, but we are running
4.8.7...

Any ideas on whether it's still the same deadlock issue or something else?

Thanks,
Charles

Mime
View raw message