lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: 'waiting on condition' caused by DirectUpdateHandler2.commit - Solr-1.3
Date Tue, 10 May 2011 12:31:12 GMT
Hmm, one thread is actively running a query (stuck building up field
cache), while another thread is waiting to acquire lock to commit?
Are you sure this is deadlock, vs it taks a really really long time to
init the field cache for this one query?

Also, you might want to try the tiny patch on
https://issues.apache.org/jira/browse/SOLR-2342 -- this gives commit
"priority" over threads just doing searching, when it tries to acquire
its lock.

Mike

http://blog.mikemccandless.com

On Mon, May 9, 2011 at 4:42 PM, Bernhard C Gass <bgass@apple.com> wrote:
> I found locking issues reported against 1.2 and 1.4 on the mailing lists, but not against
1.3.
>
> We are running five Solr-1.3 shards under OSX on Tomcat under JDK-1.6 without any problems
for quite a while. We copied the instance over to a Linux environment and now observe occasional
deadlocks.
>
> This shard would lock and never recover. Each time a different shard would would lock.
>
> Any ideas or recommendations?
>
>
> Regards,
> Bernhard
>
>
> Below the stack trace:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.2-b04 mixed mode):
>
> "http-12003-2" daemon prio=10 tid=0x000000005c45f800 nid=0x573 in Object.wait() [0x0000000042391000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaaea0bf008> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
>        at java.lang.Object.wait(Object.java:485)
>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
>        - locked <0x00002aaaea0bf008> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
>        at java.lang.Thread.run(Thread.java:619)
>
> "http-12003-1" daemon prio=10 tid=0x000000005c714000 nid=0x50c9 waiting on condition
[0x000000004228f000]
>   java.lang.Thread.State: WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0x00002aaae19f2238> (a java.util.concurrent.FutureTask$Sync)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:389)
>        at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
>        at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:104)
>        at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:113)
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>        at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
>        at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
>        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
>        at java.lang.Thread.run(Thread.java:619)
>
> "http-12003-Acceptor-0" daemon prio=10 tid=0x00002aab181de800 nid=0x5088 runnable [0x0000000041831000]
>   java.lang.Thread.State: RUNNABLE
>        at java.net.PlainSocketImpl.socketAccept(Native Method)
>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>        - locked <0x00002aaac6b0e4c0> (a java.net.SocksSocketImpl)
>        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>        at java.net.ServerSocket.accept(ServerSocket.java:421)
>        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
>        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:317)
>        at java.lang.Thread.run(Thread.java:619)
>
> "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00002aab180d2000
nid=0x5087 runnable [0x0000000041db1000]
>   java.lang.Thread.State: RUNNABLE
>        at java.lang.StringCoding.set(StringCoding.java:53)
>        at java.lang.StringCoding.encode(StringCoding.java:270)
>        at java.lang.String.getBytes(String.java:946)
>        at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
>        at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
>        at java.io.File.exists(File.java:733)
>        at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1055)
>        - locked <0x00002aaac59cab28> (a org.apache.catalina.startup.HostConfig)
>        at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271)
>        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296)
>        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
>        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
>        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
>        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
>        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
>        at java.lang.Thread.run(Thread.java:619)
>
> "pool-3-thread-1" prio=10 tid=0x00002aab1824f000 nid=0x5080 waiting on condition [0x0000000040c43000]
>   java.lang.Thread.State: RUNNABLE
>        at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
>        at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
>        at org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(TermInfosReader.java:177)
>        - locked <0x00002aaae19f0790> (a org.apache.lucene.index.TermInfosReader)
>        at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:224)
>        at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:331)
>        at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:709)
>        at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.<init>(MultiSegmentReader.java:463)
>        at org.apache.lucene.index.MultiSegmentReader.terms(MultiSegmentReader.java:368)
>        at org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:363)
>        at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
>        - locked <0x00002aaadf827cd8> (a org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder)
>        at org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352)
>        at org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSortedHitQueue.java:416)
>        at org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:207)
>        at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
>        - locked <0x00002aaadf827b70> (a org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder)
>        at org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:168)
>        at org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56)
>        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:907)
>        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
>        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1228)
>        at org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
>        at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1051)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
>
> "pool-1-thread-1" prio=10 tid=0x00002aab18222800 nid=0x507d waiting on condition [0x000000004154b000]
>   java.lang.Thread.State: WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0x00002aaac5ac21e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:619)
>
> "Low Memory Detector" daemon prio=10 tid=0x000000005c30d000 nid=0x5074 runnable [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x000000005c30b000 nid=0x5070 waiting on condition
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x000000005c306000 nid=0x5069 waiting on condition
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x000000005c304000 nid=0x5063 runnable [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x000000005c2df800 nid=0x505d in Object.wait() [0x0000000041aaf000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>        - locked <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x000000005c2dd800 nid=0x5059 in Object.wait()
[0x00000000419ae000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock)
>        at java.lang.Object.wait(Object.java:485)
>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>        - locked <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x000000005c27d800 nid=0x503a runnable [0x0000000040895000]
>   java.lang.Thread.State: RUNNABLE
>        at java.net.PlainSocketImpl.socketAccept(Native Method)
>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>        - locked <0x00002aaac90350a8> (a java.net.SocksSocketImpl)
>        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>        at java.net.ServerSocket.accept(ServerSocket.java:421)
>        at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>        at org.apache.catalina.startup.Catalina.await(Catalina.java:647)
>        at org.apache.catalina.startup.Catalina.start(Catalina.java:607)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>
> "VM Thread" prio=10 tid=0x000000005c2d9000 nid=0x5056 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c290800 nid=0x5047 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c292000 nid=0x5050 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c294000 nid=0x5051 runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c296000 nid=0x5052 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x000000005c310000 nid=0x5078 waiting on condition
>
> JNI global references: 1203
>
> Heap
>  PSYoungGen      total 431360K, used 409536K [0x00002aaaf3840000, 0x00002aab13840000,
0x00002aab13840000)
>  eden space 409536K, 100% used [0x00002aaaf3840000,0x00002aab0c830000,0x00002aab0c830000)
>  from space 21824K, 0% used [0x00002aab122f0000,0x00002aab122f0000,0x00002aab13840000)
>  to   space 21824K, 0% used [0x00002aab10da0000,0x00002aab10da0000,0x00002aab122f0000)
>  PSOldGen        total 1048576K, used 893874K [0x00002aaab3840000, 0x00002aaaf3840000,
0x00002aaaf3840000)
>  object space 1048576K, 85% used [0x00002aaab3840000,0x00002aaaea12cb28,0x00002aaaf3840000)
>  PSPermGen       total 21248K, used 18913K [0x00002aaaae440000, 0x00002aaaaf900000,
0x00002aaab3840000)
>  object space 21248K, 89% used [0x00002aaaae440000,0x00002aaaaf6b8438,0x00002aaaaf900000)
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message