lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noble Paul നോബിള്‍ नोब्ळ् <noble.p...@corp.aol.com>
Subject Re: <no subject> aka Replication Stall
Date Thu, 07 May 2009 06:28:24 GMT
hi Jeff,
Did you observe a hang even after applying SOLR-1096
(https://issues.apache.org/jira/browse/SOLR-1096) ? This change gets
applied only on the slave . So you would get the benefit only if the
slave is updated with a new build


On Tue, May 5, 2009 at 9:12 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
> It does still stall.  This time I have included the master server thread
> dump.
>
> SOLR
> Solr Amin ()
> Thread Dump
> 6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
> Thread Count: current=27, peak=29, daemon=23
> Full Thread Dump:
> 'http-8080-exec-13' Id=36, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
> a sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
> user time=1010960.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> t
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-exec-11' Id=34, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
> d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-10' Id=33, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
> d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
> at sun.mis.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbsractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.ang.Thread.run(Thread.java:619)
> 'http-8080-exec-9' Id=32, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
> at sun.misc.Unsaf.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-8' Id=29, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
> d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-7' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=26, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-5' Id=24, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-4' Id=23, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
> at sun.miscUnsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-3' Id=22, TIMED_WAITING on
> lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-2' Id=21, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-1' Id=19, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> tim=6480.0000ms user time=2900.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
> time=83050.0000ms user time=36360.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
> user time=11940.0000s
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
> TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-5-thread-1' Id=13, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
> 5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-3-thread-1' Id=12, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
> 5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-1-thread-1' Id=11, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
> 6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
> time=5580.0000ms user time=2120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
> time=333280.0000ms user time=107520.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
> total cpu time=4900.0000ms user time=940.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
> time=440.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
> time=11180.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Return to Admin Page
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Yonik Seeley <yonik@lucidimagination.com>
>> Reply-To: <solr-user@lucene.apache.org>
>> Date: Mon, 4 May 2009 16:07:50 -0400
>> To: <solr-user@lucene.apache.org>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
>>> Absolutely.  I have been monitoring but it seems to only happen when a full
>>> reindex occurs and even then intermittently.
>>
>> Any update on this?  Have you seen any additional hangs and been able
>> to get a thread dump of the master?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>>
>>>
>>>> From: Yonik Seeley <yonik@lucidimagination.com>
>>>> Reply-To: <solr-user@lucene.apache.org>
>>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>>> To: <solr-user@lucene.apache.org>
>>>> Subject: Re: <no subject> aka Replication Stall
>>>>
>>>> Next time a hang happens, could you get a thread dump of the master also?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jnewburn@zappos.com> wrote:
>>>>> I apologize for the delay.  The replication stalling out doesn't happen
>>>>> daily.  I will paste the thread dump below to try to help.  This is on a
>>>>> server that is currently locked on replication for a few hours.  Any more
>>>>> information please let me know. There are no errors in the logs either so
>>>>> very strange.
>>>>>
>>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>>> Thread Count: current=73, peak=125, daemon=70
>>>>> Full Thread Dump:
>>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>>> user time=23940.0000ms
>>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>> at
>>>>>
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
> v
>>>>> a:182)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>>> at
>>>>>
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
> m
>>>>> .java:108)
>>>>> at
>>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>>> at
>>>>>
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
> 1
>>>>> 22)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
> :
>>>>> 808)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
> 4
>>>>> )
>>>>> at
>>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>>> at
>>>>>
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
> v
>>>>> a:224)
>>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
> $
>>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
> i
>>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
> h
>>>>> eduledThreadPoolExecutor.java:205)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>>> time=1800610.0000ms user time=1743610.0000ms
>>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
> 3
>>>>> 74)
>>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:290)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
> .
>>>>> java:630)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
> s
>>>>> patcher.java:436)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
> h
>>>>> er.java:374)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
> r
>>>>> .java:302)
>>>>> at
>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
> :
>>>>> 273)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:235)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
> a
>>>>> va:233)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
> a
>>>>> va:175)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
> 8
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
> 2
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
> a
>>>>> :109)
>>>>> at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
> :
>>>>> 879)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
> H
>>>>> ttp11NioProtocol.java:719)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
> :
>>>>> 2080)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>>> user time=0.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at
>>>>>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
> u
>>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>>> time=31410.0000ms user time=24650.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>>> user time=250.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>>> time=764410.0000ms user time=394670.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>>> time=128740.0000ms user time=95550.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
> t
>>>>> ainerBase.java:1579)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>>> time=17960.0000ms user time=11850.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>>> time=261720.0000ms user time=120120.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
> r
>>>>> SocketFactory.java:34)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>>> user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> 'Finalizer' Id=3, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>>> time=16460.0000ms
>>>>> user time=15430.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>> 'Reference Handler' Id=2, WAITING on
>>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>>> time=3390.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:485)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>>> time=1570.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
> 9
>>>>> )
>>>>> at
>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
> l
>>>>> .java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>>
>>>>>> From: Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@gmail.com>
>>>>>> Reply-To: <solr-user@lucene.apache.org>
>>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>>> To: <solr-user@lucene.apache.org>
>>>>>> Subject: Re: <no subject>
>>>>>>
>>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>>> snappuller.
>>>>>>
>>>>>> I guess it should be set to some very high value say 1hr for
>>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>>> configurable .
>>>>>>
>>>>>> --Noble
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>>> <shalinmangar@gmail.com> wrote:
>>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>>> we
>>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>>> <yonik@lucidimagination.com>wrote:
>>>>>>>
>>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Yonik
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jnewburn@zappos.com>
>>>>>>>> wrote:
>>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>>> nightly
>>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>>>>> will
>>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>>> to
>>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>>> always
>>>>>>>>> the same server.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>>
>>>>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>>>>> Poll Interval     00:05:00
>>>>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>>>>     Size: 1.29 GB
>>>>>>>>>     Times Replicated Since Startup: 3591
>>>>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>>>     Files Downloaded: 12 / 163
>>>>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>>> 163
>>>>>>>>> bytes/s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Newburn
>>>>>>>>> Software Engineer, Zappos.com
>>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>
>>>>>
>>>
>>>
>
>



-- 
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

Mime
View raw message