lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tomlewlit (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SOLR-9871) Java-level deadlock on SolrCore.getSearcher
Date Fri, 16 Dec 2016 08:55:58 GMT
tomlewlit created SOLR-9871:
-------------------------------

             Summary: Java-level deadlock on SolrCore.getSearcher
                 Key: SOLR-9871
                 URL: https://issues.apache.org/jira/browse/SOLR-9871
             Project: Solr
          Issue Type: Bug
      Security Level: Public (Default Security Level. Issues are Public)
          Components: SolrCloud
    Affects Versions: 6.2.1
            Reporter: tomlewlit


I have a SolrCloud setup with 2 nodes, 4 shards, 2 replicas, ~200mln documents. Sometimes
one of the nodes hangs, jstack tells about java-level deadlock:

{noformat}
Found one Java-level deadlock:
=============================
"qtp1543727556-8467":
  waiting to lock monitor 0x00007f5b6c9d8378 (object 0x000000050dc96730, a java.lang.Object),
  which is held by "searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2
s:shard1 c:mail r:core_node8"
"searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2 s:shard1
c:mail r:core_node8":
  waiting to lock monitor 0x00007f5aac41cd38 (object 0x000000050ded63e8, a org.apache.solr.update.SolrIndexWriter),
  which is held by "commitScheduler-21-thread-1"
"commitScheduler-21-thread-1":
  waiting to lock monitor 0x00007f5b6c9d8378 (object 0x000000050dc96730, a java.lang.Object),
  which is held by "searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2
s:shard1 c:mail r:core_node8"

Java stack information for the threads listed above:
===================================================
"qtp1543727556-8467":
	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1749)
	- waiting to lock <0x000000050dc96730> (a java.lang.Object)
	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1552)
	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1487)
	at org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:115)
	at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:308)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
"searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2 s:shard1
c:mail r:core_node8":
	at org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:4974)
	- waiting to lock <0x000000050ded63e8> (a org.apache.solr.update.SolrIndexWriter)
	at org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:381)
	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:251)
	at org.apache.solr.search.SolrIndexSearcher.close(SolrIndexSearcher.java:461)
	at org.apache.solr.core.SolrCore$2.close(SolrCore.java:1982)
	at org.apache.solr.util.RefCounted.decref(RefCounted.java:56)
	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:2016)
	- locked <0x000000050dc96730> (a java.lang.Object)
	at org.apache.solr.core.SolrCore.lambda$getSearcher$5(SolrCore.java:1903)
	at org.apache.solr.core.SolrCore$$Lambda$84/1360460105.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$3/23599925.run(Unknown
Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"commitScheduler-21-thread-1":
	at org.apache.solr.core.SolrCore.getIndexDir(SolrCore.java:302)
	- waiting to lock <0x000000050dc96730> (a java.lang.Object)
	at org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager.isSnapshotted(SolrSnapshotMetaDataManager.java:232)
	- locked <0x00000004d38deaa0> (a org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager)
	at org.apache.solr.core.IndexDeletionPolicyWrapper$IndexCommitWrapper.delete(IndexDeletionPolicyWrapper.java:195)
	at org.apache.solr.core.SolrDeletionPolicy.updateCommits(SolrDeletionPolicy.java:199)
	- locked <0x0000000519863980> (a org.apache.solr.core.SolrDeletionPolicy)
	at org.apache.solr.core.SolrDeletionPolicy.onCommit(SolrDeletionPolicy.java:101)
	at org.apache.solr.core.IndexDeletionPolicyWrapper.onCommit(IndexDeletionPolicyWrapper.java:161)
	at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:526)
	at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3252)
	- locked <0x000000050ded63e8> (a org.apache.solr.update.SolrIndexWriter)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3214)
	- locked <0x000000050deed630> (a java.lang.Object)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3171)
	at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:607)
	at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message