geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiram Chirino" <hi...@hiramchirino.com>
Subject Re: ActiveMQ 3.2.4-SNAPSHOT bug causes Geronimo 1.1 shutdown to hang.
Date Wed, 05 Apr 2006 18:00:18 GMT
Hi,

I think I see the issue.  Working on the fix now.


On 4/4/06, Kevan Miller <kevan.miller@gmail.com> wrote:
>
> On Apr 4, 2006, at 10:07 AM, John Sisson wrote:
>
> > Kevin, I tried:
> >
> >    export GERONIMO_OPTS=-Dactivemq.broker.disable-clean-shutdown=true
> >    ./geronimo.sh run --long
> >
> > The problem still occurred.  I'll investigate further.
> >
> > AFAIK, the EOFException warning message just needs to be changed to
> > a trace level message call.  Hiram can you confirm?
>
> Juhn,
> Thanks.
>
> It's debatable rather or not the warning message should be a trace
> level message. I'll leave that for the AMQ guys...
>
> Both sets of problems indicate that the AMQ Broker is being stopped
> prior to the stopping of the AMQ clients in the Geronimo server.
>
> There's a third potential explanation -- the order of shutdown
> processing within Geronimo has been changed. The kernel is stopping
> the broker prior to stopping the clients...
>
> --kevan
>
> >
> > John
> >
> > Kevan Miller wrote:
> >> John,
> >> We seem to have lost a work-around that avoided a number of
> >> ActiveMQ shutdown problems (including the one you're butting your
> >> head against). FYI, I've also noticed the following on a ctrl-c of
> >> a 1.1 server:
> >>
> >> 12:46:57,193 WARN  [TcpTransportChannel] Peer closed connection
> >> java.io.EOFException
> >>     at java.io.DataInputStream.readByte(DataInputStream.java:333)
> >>     at org.activemq.io.AbstractWireFormat.readPacket
> >> (AbstractWireFormat.java:230)
> >>     at org.activemq.transport.tcp.TcpTransportChannel.run
> >> (TcpTransportChannel.java:313)
> >>     at java.lang.Thread.run(Thread.java:552)
> >>
> >> We had a work-around for these problems in 1.0 by disabling the
> >> ActiveMQ shutdown hooks, with the following in the ActiveMQ broker
> >> config plan:
> >>
> >>     <gbean name="SystemProperties"
> >> class="org.apache.geronimo.system.properties.SystemProperties">
> >>         <attribute name="systemProperties">
> >>             activemq.broker.disable-clean-shutdown=true
> >>         </attribute>
> >>     </gbean>
> >>
> >> The SystemProperties GBean is still in the ActiveMQ plan in 1.1.
> >> So, either ActiveMQ no longer honors the setting or the
> >> SystemProperties GBean isn't working. I'd guess that the GBean is
> >> broken, but probably best to test ActiveMQ using '-
> >> Dactivemq.broker.disable-clean-shutdown=true'. Could you give it a
> >> whirl?
> >>
> >> FYI, the deadlock condition is already reported as a problem:
> >>
> >> http://issues.apache.org/jira/browse/GERONIMO-1493
> >>
> >> Hiram created the following jira to see if the problem has been
> >> fixed on AMQ v4 rc2
> >>
> >> http://jira.activemq.org/jira/browse/AMQ-651
> >>
> >> --kevan
> >>
> >> On Apr 3, 2006, at 9:21 PM, John Sisson wrote:
> >>
> >>> I have reproduced this twice in a row on Solaris x86.
> >>> Can someone with access please create a version record in JIRA
> >>> for ActiveMQ 3.2.4 so we can raise issues against it.
> >>>
> >>> See stack dump below:
> >>>
> >>> $./geronimo.sh run --long
> >>> Using GERONIMO_BASE:   /home/sissonj/OpenSourceJava/asf/geronimo/
> >>> branches/1.1/assemblies/j2ee-tomcat-server/target/geronimo-1.1-
> >>> SNAPSHOT
> >>> Using GERONIMO_HOME:   /home/sissonj/OpenSourceJava/asf/geronimo/
> >>> branches/1.1/assemblies/j2ee-tomcat-server/target/geronimo-1.1-
> >>> SNAPSHOT
> >>> Using GERONIMO_TMPDIR: /home/sissonj/OpenSourceJava/asf/geronimo/
> >>> branches/1.1/assemblies/j2ee-tomcat-server/target/geronimo-1.1-
> >>> SNAPSHOT/var/temp
> >>> Using JRE_HOME:        /usr/j2se
> >>> Booting Geronimo Kernel (in Java 1.4.2_10)...
> >>> Configuration geronimo/rmi-naming/1.1-SNAPSHOT/car started in
> >>> 1/22   0s
> >>> Configuration geronimo/j2ee-server/1.1-SNAPSHOT/car started in
> >>> 2/22   1s
> >>> Configuration geronimo/j2ee-security/1.1-SNAPSHOT/car started in
> >>> 3/22   1s
> >>> Configuration geronimo/system-database/1.1-SNAPSHOT/car started
> >>> in  4/22   3s
> >>> Configuration geronimo/activemq-broker/1.1-SNAPSHOT/car started
> >>> in  5/22   1s
> >>> Configuration geronimo/activemq/1.1-SNAPSHOT/car started in
> >>> 6/22   0s
> >>> Configuration geronimo/directory/1.1-SNAPSHOT/car started in
> >>> 7/22   1s
> >>> Configuration geronimo/ldap-realm/1.1-SNAPSHOT/car started in
> >>> 8/22   0s
> >>> Configuration geronimo/tomcat/1.1-SNAPSHOT/car started in  9/22   3s
> >>> Configuration geronimo/geronimo-gbean-deployer/1.1-SNAPSHOT/car
> >>> started in 10/22   1s
> >>> Configuration geronimo/j2ee-deployer/1.1-SNAPSHOT/car started in
> >>> 11/22   1s
> >>> Configuration geronimo/tomcat-deployer/1.1-SNAPSHOT/car started
> >>> in 12/22   0s
> >>> Configuration geronimo/welcome-tomcat/1.1-SNAPSHOT/car started in
> >>> 13/22   0s
> >>> Configuration geronimo/ldap-demo-tomcat/1.1-SNAPSHOT/car started
> >>> in 14/22   0s
> >>> Configuration geronimo/servlets-examples-tomcat/1.1-SNAPSHOT/car
> >>> started in 15/22   0s
> >>> Configuration geronimo/jsp-examples-tomcat/1.1-SNAPSHOT/car
> >>> started in 16/22   1s
> >>> Configuration geronimo/webconsole-tomcat/1.1-SNAPSHOT/car started
> >>> in 17/22   1s
> >>> Configuration geronimo/uddi-tomcat/1.1-SNAPSHOT/car started in
> >>> 18/22   0s
> >>> Configuration geronimo/jmxdebug-tomcat/1.1-SNAPSHOT/car started
> >>> in 19/22   0s
> >>> Configuration geronimo/daytrader-derby-tomcat/1.1-SNAPSHOT/car
> >>> started in 20/22   3s
> >>> Configuration geronimo/remote-deploy-tomcat/1.1-SNAPSHOT/car
> >>> started in 21/22   0s
> >>> Configuration geronimo/hot-deployer/1.1-SNAPSHOT/
> >>> carjava.lang.IllegalArgumentException: Unable to look up
> >>> connectors for Tomcat container 'geronimo/tomcat/1.1-SNAPSHOT/car?
> >>> ServiceModule=geronimo/tomcat/1.1-SNAPSHOT/
> >>> car,j2eeType=GBean,name=TomcatWebContainer':
> >>> java.lang.NullPointerException
> >>>        at
> >>> org.apache.geronimo.tomcat.TomcatManagerImpl.getConnectorsForContain
> >>> er(TomcatManagerImpl.java:264)
> >>>        at org.apache.geronimo.tomcat.TomcatManagerImpl$
> >>> $FastClassByCGLIB$$6ff876df.invoke(<generated>)
> >>>        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
> >>> (FastMethodInvoker.java:38)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
> >>> (GBeanOperation.java:122)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
> >>> (GBeanInstance.java:848)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.invoke
> >>> (BasicKernel.java:244)
> >>>        at
> >>> org.apache.geronimo.deployment.plugin.local.CommandSupport.mapContai
> >>> nersToURLs(CommandSupport.java:371)
> >>>        at
> >>> org.apache.geronimo.deployment.plugin.local.CommandSupport.addWebURL
> >>> s(CommandSupport.java:294)
> >>>        at
> >>> org.apache.geronimo.deployment.plugin.jmx.JMXDeploymentManager.getMo
> >>> dules(JMXDeploymentManager.java:164)
> >>>        at
> >>> org.apache.geronimo.deployment.plugin.jmx.JMXDeploymentManager.getAv
> >>> ailableModules(JMXDeploymentManager.java:110)
> >>>        at
> >>> org.apache.geronimo.deployment.hot.DirectoryHotDeployer.doStart
> >>> (DirectoryHotDeployer.java:141)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance
> >>> (GBeanInstance.java:977)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStar
> >>> t(GBeanInstanceState.java:267)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.start
> >>> (GBeanInstanceState.java:102)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.startRecursive(
> >>> GBeanInstanceState.java:124)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstance.startRecursive
> >>> (GBeanInstance.java:546)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.BasicKernel.startRecursiveGBean
> >>> (BasicKernel.java:391)
> >>>        at
> >>> org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurati
> >>> onGBeans(ConfigurationUtil.java:245)
> >>>        at
> >>> org.apache.geronimo.kernel.config.KernelConfigurationManager.start
> >>> (KernelConfigurationManager.java:163)
> >>>        at
> >>> org.apache.geronimo.kernel.config.SimpleConfigurationManager.startCo
> >>> nfiguration(SimpleConfigurationManager.java:331)
> >>>        at
> >>> org.apache.geronimo.kernel.config.SimpleConfigurationManager$
> >>> $FastClassByCGLIB$$ce77a924.invoke(<generated>)
> >>>        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
> >>> (FastMethodInvoker.java:38)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
> >>> (GBeanOperation.java:122)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
> >>> (GBeanInstance.java:813)
> >>>        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke
> >>> (RawInvoker.java:57)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke
> >>> (RawOperationInvoker.java:35)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept
> >>> (ProxyMethodInterceptor.java:96)
> >>>        at org.apache.geronimo.kernel.config.ConfigurationManager$
> >>> $EnhancerByCGLIB$$1b75b2b5.startConfiguration(<generated>)
> >>>        at org.apache.geronimo.system.main.Daemon.doStartup
> >>> (Daemon.java:287)
> >>>        at org.apache.geronimo.system.main.Daemon.<init>
> >>> (Daemon.java:74)
> >>>        at org.apache.geronimo.system.main.Daemon.main(Daemon.java:
> >>> 367)
> >>> started in 22/22   1s
> >>> Startup completed in 25 seconds
> >>> java.lang.IllegalArgumentException: Unable to look up connectors
> >>> for Tomcat container 'geronimo/tomcat/1.1-SNAPSHOT/car?
> >>> ServiceModule=geronimo/tomcat/1.1-SNAPSHOT/
> >>> car,j2eeType=GBean,name=TomcatWebContainer':
> >>> java.lang.NullPointerException
> >>>        at
> >>> org.apache.geronimo.tomcat.TomcatManagerImpl.getConnectorsForContain
> >>> er(TomcatManagerImpl.java:264)
> >>>        at org.apache.geronimo.tomcat.TomcatManagerImpl$
> >>> $FastClassByCGLIB$$6ff876df.invoke(<generated>)
> >>>        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
> >>> (FastMethodInvoker.java:38)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
> >>> (GBeanOperation.java:122)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
> >>> (GBeanInstance.java:848)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.invoke
> >>> (BasicKernel.java:244)
> >>>        at
> >>> org.apache.geronimo.system.main.WebAppUtil.mapContainersToURLs
> >>> (WebAppUtil.java:52)
> >>>        at
> >>> org.apache.geronimo.system.main.LongStartupMonitor.wrapUp
> >>> (LongStartupMonitor.java:195)
> >>>        at
> >>> org.apache.geronimo.system.main.LongStartupMonitor.startupFinished
> >>> (LongStartupMonitor.java:104)
> >>>        at org.apache.geronimo.system.main.Daemon.doStartup
> >>> (Daemon.java:314)
> >>>        at org.apache.geronimo.system.main.Daemon.<init>
> >>> (Daemon.java:74)
> >>>        at org.apache.geronimo.system.main.Daemon.main(Daemon.java:
> >>> 367)
> >>>  Listening on Ports:
> >>>    1099 0.0.0.0   RMI Naming
> >>>    1389 0.0.0.0   Apache Directory LDAP
> >>>    1527 0.0.0.0   Derby Connector
> >>>    4201 127.0.0.1 ActiveIO Connector EJB
> >>>    4242 127.0.0.1 Remote Login Listener
> >>>    8009 0.0.0.0   Tomcat Connector AJP
> >>>    8080 0.0.0.0   Tomcat Connector HTTP
> >>>    8443 0.0.0.0   Tomcat Connector HTTPS
> >>>    9999 0.0.0.0   JMX Remoting Connector
> >>>   61616 0.0.0.0   ActiveMQ Message Broker Connector
> >>>
> >>>  Started Application Modules:
> >>>    WAR: geronimo/jsp-examples-tomcat/1.1-SNAPSHOT/car
> >>>    EAR: geronimo/daytrader-derby-tomcat/1.1-SNAPSHOT/car
> >>>    WAR: geronimo/welcome-tomcat/1.1-SNAPSHOT/car
> >>>    WAR: geronimo/servlets-examples-tomcat/1.1-SNAPSHOT/car
> >>>    WAR: geronimo/ldap-demo-tomcat/1.1-SNAPSHOT/car
> >>>    RAR: geronimo/activemq/1.1-SNAPSHOT/car
> >>>    WAR: geronimo/jmxdebug-tomcat/1.1-SNAPSHOT/car
> >>>    EAR: geronimo/webconsole-tomcat/1.1-SNAPSHOT/car
> >>>    EAR: geronimo/uddi-tomcat/1.1-SNAPSHOT/car
> >>>    RAR: geronimo/system-database/1.1-SNAPSHOT/car
> >>>    WAR: geronimo/remote-deploy-tomcat/1.1-SNAPSHOT/car
> >>>
> >>>  Web Applications:
> >>>    null/
> >>>    null/console
> >>>    null/console-standard
> >>>    null/daytrader
> >>>    null/debug-tool
> >>>    null/jsp-examples
> >>>    null/juddi
> >>>    null/ldap-demo
> >>>    null/remote-deploy
> >>>    null/servlets-examples
> >>>
> >>> Geronimo Application Server started
> >>> Server shutdown begun
> >>> 09:30:44,124 WARN  [TcpTransportChannel] Peer closed connection
> >>> java.io.EOFException
> >>>        at java.io.DataInputStream.readByte(DataInputStream.java:333)
> >>>        at org.activemq.io.AbstractWireFormat.readPacket
> >>> (AbstractWireFormat.java:230)
> >>>        at org.activemq.transport.tcp.TcpTransportChannel.run
> >>> (TcpTransportChannel.java:313)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>> 09:30:44,130 WARN  [TcpTransportChannel] Peer closed connection
> >>> java.io.EOFException
> >>>        at java.io.DataInputStream.readByte(DataInputStream.java:333)
> >>>        at org.activemq.io.AbstractWireFormat.readPacket
> >>> (AbstractWireFormat.java:230)
> >>>        at org.activemq.transport.tcp.TcpTransportChannel.run
> >>> (TcpTransportChannel.java:313)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>> ^CFull thread dump Java HotSpot(TM) Client VM (1.4.2_10-b03 mixed
> >>> mode):
> >>>
> >>> "Thread-94" prio=7 tid=0x08706748 nid=0x91 waiting on condition
> >>> [c41d5000..c41d5d98]
> >>>        at java.lang.Thread.sleep(Native Method)
> >>>        at org.activemq.ra.ActiveMQAsfEndpointWorker.reconnect
> >>> (ActiveMQAsfEndpointWorker.java:177)
> >>>        - locked <0xc7711290> (a
> >>> org.activemq.ra.ActiveMQAsfEndpointWorker)
> >>>        at org.activemq.ra.ActiveMQAsfEndpointWorker.access$200
> >>> (ActiveMQAsfEndpointWorker.java:40)
> >>>        at org.activemq.ra.ActiveMQAsfEndpointWorker$1.run
> >>> (ActiveMQAsfEndpointWorker.java:105)
> >>>        - locked <0xc76fcc30> (a
> >>> org.activemq.ra.ActiveMQAsfEndpointWorker$1)
> >>>        at org.apache.geronimo.connector.work.WorkerContext.run
> >>> (WorkerContext.java:291)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor
> >>> $Worker.run(Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-93" prio=7 tid=0x08771bd8 nid=0x90 in Object.wait()
> >>> [c09fd000..c09fdd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll
> >>> (Unknown Source)
> >>>        - locked <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask
> >>> (Unknown Source)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor
> >>> $Worker.run(Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-92" prio=7 tid=0x08250a20 nid=0x8f in Object.wait()
> >>> [c349d000..c349dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll
> >>> (Unknown Source)
> >>>        - locked <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask
> >>> (Unknown Source)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor
> >>> $Worker.run(Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-91" prio=7 tid=0x0929ba18 nid=0x8e in Object.wait()
> >>> [c37ab000..c37abd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll
> >>> (Unknown Source)
> >>>        - locked <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask
> >>> (Unknown Source)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor
> >>> $Worker.run(Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-90" prio=7 tid=0x0853b4c8 nid=0x8d in Object.wait()
> >>> [c37ed000..c37edd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll
> >>> (Unknown Source)
> >>>        - locked <0xc64a7e28> (a java.lang.Object)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask
> >>> (Unknown Source)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor
> >>> $Worker.run(Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "SIGINT handler" daemon prio=10 tid=0x0886ea08 nid=0x8a waiting
> >>> for monitor entry [c4193000..c4193d98]
> >>>        at java.lang.Shutdown.exit(Shutdown.java:210)
> >>>        - waiting to lock <0xca095c18> (a java.lang.Class)
> >>>        at java.lang.Terminator$1.handle(Terminator.java:35)
> >>>        at sun.misc.Signal$1.run(Signal.java:195)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Geronimo shutdown thread" prio=5 tid=0x0876fe50 nid=0x78 waiting
> >>> for monitor entry [c09bb000..c09bbd98]
> >>>        at org.activemq.ra.ActiveMQAsfEndpointWorker.stop
> >>> (ActiveMQAsfEndpointWorker.java:135)
> >>>        - waiting to lock <0xc7711290> (a
> >>> org.activemq.ra.ActiveMQAsfEndpointWorker)
> >>>        at
> >>> org.activemq.ra.ActiveMQResourceAdapter.endpointDeactivation
> >>> (ActiveMQResourceAdapter.java:261)
> >>>        at
> >>> org.apache.geronimo.connector.ResourceAdapterWrapper.endpointDeactiv
> >>> ation(ResourceAdapterWrapper.java:92)
> >>>        at org.apache.geronimo.connector.ResourceAdapterWrapper$
> >>> $FastClassByCGLIB$$4ab28e73.invoke(<generated>)
> >>>        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
> >>> (FastMethodInvoker.java:38)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
> >>> (GBeanOperation.java:122)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
> >>> (GBeanInstance.java:813)
> >>>        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke
> >>> (RawInvoker.java:57)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke
> >>> (RawOperationInvoker.java:35)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept
> >>> (ProxyMethodInterceptor.java:96)
> >>>        at org.apache.geronimo.connector.ResourceAdapterWrapper$
> >>> $EnhancerByCGLIB$$d7c17cb8.endpointDeactivation(<generated>)
> >>>        at
> >>> org.apache.geronimo.connector.ActivationSpecWrapper.deactivate
> >>> (ActivationSpecWrapper.java:109)
> >>>        at org.apache.geronimo.connector.ActivationSpecWrapper$
> >>> $FastClassByCGLIB$$aaa078c1.invoke(<generated>)
> >>>        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
> >>> (FastMethodInvoker.java:38)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
> >>> (GBeanOperation.java:122)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
> >>> (GBeanInstance.java:813)
> >>>        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke
> >>> (RawInvoker.java:57)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke
> >>> (RawOperationInvoker.java:35)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept
> >>> (ProxyMethodInterceptor.java:96)
> >>>        at org.apache.geronimo.connector.ActivationSpecWrapper$
> >>> $EnhancerByCGLIB$$d908fdf6.deactivate(<generated>)
> >>>        at org.openejb.mdb.MDBContainer.doStop(MDBContainer.java:226)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstance.destroyInstance
> >>> (GBeanInstance.java:1139)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStop
> >>> (GBeanInstanceState.java:337)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop
> >>> (GBeanInstanceState.java:188)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.stop
> >>> (GBeanInstance.java:554)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean
> >>> (BasicKernel.java:435)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop
> >>> (GBeanInstanceState.java:180)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.stop
> >>> (GBeanInstance.java:554)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean
> >>> (BasicKernel.java:435)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop
> >>> (GBeanInstanceState.java:180)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.stop
> >>> (GBeanInstance.java:554)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean
> >>> (BasicKernel.java:435)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop
> >>> (GBeanInstanceState.java:180)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.stop
> >>> (GBeanInstance.java:554)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean
> >>> (BasicKernel.java:435)
> >>>        at
> >>> org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop
> >>> (GBeanInstanceState.java:180)
> >>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.stop
> >>> (GBeanInstance.java:554)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean
> >>> (BasicKernel.java:435)
> >>>        at
> >>> org.apache.geronimo.kernel.config.KernelConfigurationManager
> >>> $ShutdownHook.run(KernelConfigurationManager.java:261)
> >>>        at
> >>> org.apache.geronimo.kernel.basic.BasicKernel.notifyShutdownHooks
> >>> (BasicKernel.java:685)
> >>>        at org.apache.geronimo.kernel.basic.BasicKernel.shutdown
> >>> (BasicKernel.java:662)
> >>>        at org.apache.geronimo.system.main.Daemon$1.run
> >>> (Daemon.java:245)
> >>>
> >>> "SIGINT handler" daemon prio=10 tid=0x09240c30 nid=0x75 in
> >>> Object.wait() [c440b000..c440bd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc631b018> (a
> >>> org.apache.geronimo.system.main.Daemon$1)
> >>>        at java.lang.Thread.join(Thread.java:1001)
> >>>        - locked <0xc631b018> (a
> >>> org.apache.geronimo.system.main.Daemon$1)
> >>>        at java.lang.Thread.join(Thread.java:1054)
> >>>        at java.lang.Shutdown.runHooks(Shutdown.java:130)
> >>>        at java.lang.Shutdown.sequence(Shutdown.java:165)
> >>>        at java.lang.Shutdown.exit(Shutdown.java:210)
> >>>        - locked <0xca095c18> (a java.lang.Class)
> >>>        at java.lang.Terminator$1.handle(Terminator.java:35)
> >>>        at sun.misc.Signal$1.run(Signal.java:195)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-74" daemon prio=5 tid=0x0824fbb0 nid=0x73 waiting on
> >>> condition [c093d000..c093dd98]
> >>>        at java.lang.Thread.sleep(Native Method)
> >>>        at org.apache.geronimo.transaction.manager.TransactionTimer
> >>> $CurrentTime.run(TransactionTimer.java:38)
> >>>
> >>> "Geronimo Config Store Reaper" daemon prio=5 tid=0x0816bf18
> >>> nid=0x67 waiting on condition [c1edd000..c1eddd98]
> >>>        at java.lang.Thread.sleep(Native Method)
> >>>        at org.apache.geronimo.deployment.Deployer
> >>> $DeployerReaper.run(Deployer.java:365)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "TP-Monitor" daemon prio=5 tid=0x0848ec38 nid=0x66 in Object.wait
> >>> () [c1f9b000..c1f9bd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6b393d0> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $MonitorRunnable.run(ThreadPool.java:559)
> >>>        - locked <0xc6b393d0> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "TP-Processor4" daemon prio=5 tid=0x0848dbe8 nid=0x65 runnable
> >>> [c1fdd000..c1fddd98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc6b16508> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at java.net.ServerSocket.accept(ServerSocket.java:419)
> >>>        at org.apache.jk.common.ChannelSocket.accept
> >>> (ChannelSocket.java:295)
> >>>        at org.apache.jk.common.ChannelSocket.acceptConnections
> >>> (ChannelSocket.java:641)
> >>>        at org.apache.jk.common.SocketAcceptor.runIt
> >>> (ChannelSocket.java:852)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:684)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "TP-Processor3" daemon prio=5 tid=0x0848d1a0 nid=0x64 in
> >>> Object.wait() [c201f000..c201fd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6b39580> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6b39580> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "TP-Processor2" daemon prio=5 tid=0x0848c758 nid=0x63 in
> >>> Object.wait() [c2061000..c2061d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6b39600> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6b39600> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "TP-Processor1" daemon prio=5 tid=0x0848c510 nid=0x62 in
> >>> Object.wait() [c20a3000..c20a3d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6b39680> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6b39680> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Monitor" prio=5 tid=0x080dee20 nid=0x61 in
> >>> Object.wait() [c20e5000..c20e5d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a57cf8> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $MonitorRunnable.run(ThreadPool.java:559)
> >>>        - locked <0xc6a57cf8> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor25" daemon prio=5 tid=0x081427d8
> >>> nid=0x60 runnable [c2127000..c2127d98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc6a0e6b0> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at java.net.ServerSocket.accept(ServerSocket.java:419)
> >>>        at
> >>> org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket
> >>> (DefaultServerSocketFactory.java:60)
> >>>        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket
> >>> (PoolTcpEndpoint.java:407)
> >>>        at
> >>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt
> >>> (LeaderFollowerWorkerThread.java:70)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:684)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor24" daemon prio=5 tid=0x08141d90
> >>> nid=0x5f in Object.wait() [c2169000..c2169d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a57e00> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a57e00> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor23" daemon prio=5 tid=0x08141348
> >>> nid=0x5e in Object.wait() [c21ab000..c21abd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a57e80> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a57e80> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor22" daemon prio=5 tid=0x08140900
> >>> nid=0x5d in Object.wait() [c21ed000..c21edd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a57f00> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a57f00> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor21" daemon prio=5 tid=0x0813feb8
> >>> nid=0x5c in Object.wait() [c222f000..c222fd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a57f80> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a57f80> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor20" daemon prio=5 tid=0x0813f470
> >>> nid=0x5b in Object.wait() [c2271000..c2271d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58000> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58000> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor19" daemon prio=5 tid=0x0813ea28
> >>> nid=0x5a in Object.wait() [c22b3000..c22b3d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58080> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58080> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor18" daemon prio=5 tid=0x0813dfe0
> >>> nid=0x59 in Object.wait() [c22f5000..c22f5d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58100> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58100> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor17" daemon prio=5 tid=0x0813d598
> >>> nid=0x58 in Object.wait() [c2337000..c2337d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58180> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58180> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor16" daemon prio=5 tid=0x0813cb50
> >>> nid=0x57 in Object.wait() [c2379000..c2379d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58200> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58200> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor15" daemon prio=5 tid=0x0813c108
> >>> nid=0x56 in Object.wait() [c23bb000..c23bbd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58280> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58280> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor14" daemon prio=5 tid=0x0813b6c0
> >>> nid=0x55 in Object.wait() [c23fd000..c23fdd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58300> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58300> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor13" daemon prio=5 tid=0x0883a1e8
> >>> nid=0x54 in Object.wait() [c243f000..c243fd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58380> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58380> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor12" daemon prio=5 tid=0x08839198
> >>> nid=0x53 in Object.wait() [c2481000..c2481d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58400> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58400> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor11" daemon prio=5 tid=0x08838750
> >>> nid=0x52 in Object.wait() [c24c3000..c24c3d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58480> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58480> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor10" daemon prio=5 tid=0x08837d08
> >>> nid=0x51 in Object.wait() [c2505000..c2505d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58500> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58500> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor9" daemon prio=5 tid=0x088372c0
> >>> nid=0x50 in Object.wait() [c2547000..c2547d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58580> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58580> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor8" daemon prio=5 tid=0x08836878
> >>> nid=0x4f in Object.wait() [c2589000..c2589d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58600> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58600> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor7" daemon prio=5 tid=0x08835e30
> >>> nid=0x4e in Object.wait() [c25cb000..c25cbd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58680> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58680> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor6" daemon prio=5 tid=0x088353e8
> >>> nid=0x4d in Object.wait() [c260d000..c260dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58700> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58700> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor5" daemon prio=5 tid=0x088349a0
> >>> nid=0x4c in Object.wait() [c264f000..c264fd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58780> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58780> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor4" daemon prio=5 tid=0x08833f58
> >>> nid=0x4b in Object.wait() [c2691000..c2691d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58800> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58800> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor3" daemon prio=5 tid=0x08833510
> >>> nid=0x4a in Object.wait() [c26d3000..c26d3d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58880> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58880> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor2" daemon prio=5 tid=0x082f5b08
> >>> nid=0x49 in Object.wait() [c2715000..c2715d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58900> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58900> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8080-Processor1" daemon prio=5 tid=0x082f58c0
> >>> nid=0x48 in Object.wait() [c2757000..c2757d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6a58980> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc6a58980> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Monitor" prio=5 tid=0x082f26e0 nid=0x47 in
> >>> Object.wait() [c2799000..c2799d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69ed640> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $MonitorRunnable.run(ThreadPool.java:559)
> >>>        - locked <0xc69ed640> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor25" daemon prio=5 tid=0x082f1c98
> >>> nid=0x46 runnable [c27db000..c27dbd98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc69b4e58> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept
> >>> (DashoA12275)
> >>>        at
> >>> org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket
> >>> (JSSESocketFactory.java:110)
> >>>        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket
> >>> (PoolTcpEndpoint.java:407)
> >>>        at
> >>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt
> >>> (LeaderFollowerWorkerThread.java:70)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:684)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor24" daemon prio=5 tid=0x082f0848
> >>> nid=0x45 in Object.wait() [c281d000..c281dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f11b0> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f11b0> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor23" daemon prio=5 tid=0x082efe00
> >>> nid=0x44 in Object.wait() [c288f000..c288fd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f12a8> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f12a8> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor22" daemon prio=5 tid=0x082ef3b8
> >>> nid=0x43 in Object.wait() [c28d1000..c28d1d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f13a0> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f13a0> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor21" daemon prio=5 tid=0x082ee970
> >>> nid=0x42 in Object.wait() [c2913000..c2913d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1498> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1498> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor20" daemon prio=5 tid=0x088adb70
> >>> nid=0x41 in Object.wait() [c2955000..c2955d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1590> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1590> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor19" daemon prio=5 tid=0x088ad528
> >>> nid=0x40 in Object.wait() [c2997000..c2997d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1688> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1688> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor18" daemon prio=5 tid=0x088acae0
> >>> nid=0x3f in Object.wait() [c29d9000..c29d9d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1780> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1780> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor17" daemon prio=5 tid=0x088ac098
> >>> nid=0x3e in Object.wait() [c2a1b000..c2a1bd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1878> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1878> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor16" daemon prio=5 tid=0x088ab650
> >>> nid=0x3d in Object.wait() [c2a5d000..c2a5dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1970> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1970> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor15" daemon prio=5 tid=0x088aac08
> >>> nid=0x3c in Object.wait() [c2a9f000..c2a9fd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1a68> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1a68> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor14" daemon prio=5 tid=0x088a9bb8
> >>> nid=0x3b in Object.wait() [c2ae1000..c2ae1d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1b60> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1b60> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor13" daemon prio=5 tid=0x088a9170
> >>> nid=0x3a in Object.wait() [c2b23000..c2b23d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1c58> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1c58> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor12" daemon prio=5 tid=0x088a8728
> >>> nid=0x39 in Object.wait() [c2b65000..c2b65d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1d50> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1d50> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor11" daemon prio=5 tid=0x088a7ce0
> >>> nid=0x38 in Object.wait() [c2ba7000..c2ba7d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1e48> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1e48> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor10" daemon prio=5 tid=0x088a7298
> >>> nid=0x37 in Object.wait() [c2be9000..c2be9d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f1f40> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f1f40> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor9" daemon prio=5 tid=0x088a6930
> >>> nid=0x36 in Object.wait() [c2c2b000..c2c2bd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2038> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2038> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor8" daemon prio=5 tid=0x0830f658
> >>> nid=0x35 in Object.wait() [c2c6d000..c2c6dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2130> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2130> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor7" daemon prio=5 tid=0x0830f0f0
> >>> nid=0x34 in Object.wait() [c2caf000..c2cafd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2228> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2228> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor6" daemon prio=5 tid=0x084c8ec0
> >>> nid=0x33 in Object.wait() [c2cf1000..c2cf1d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2320> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2320> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor5" daemon prio=5 tid=0x084c8d58
> >>> nid=0x32 in Object.wait() [c2d33000..c2d33d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2418> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2418> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor4" daemon prio=5 tid=0x08310248
> >>> nid=0x31 in Object.wait() [c2d75000..c2d75d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2510> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2510> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor3" daemon prio=5 tid=0x084c9898
> >>> nid=0x30 in Object.wait() [c2db7000..c2db7d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2608> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2608> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor2" daemon prio=5 tid=0x084c9730
> >>> nid=0x2f in Object.wait() [c2df9000..c2df9d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f2700> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f2700> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "http-0.0.0.0-8443-Processor1" daemon prio=5 tid=0x0830f8e0
> >>> nid=0x2e in Object.wait() [c2e3b000..c2e3bd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc69f27f8> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.tomcat.util.threads.ThreadPool
> >>> $ControlRunnable.run(ThreadPool.java:656)
> >>>        - locked <0xc69f27f8> (a
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "ContainerBackgroundProcessor[StandardEngine[Geronimo]]" daemon
> >>> prio=5 tid=0x082b85c0 nid=0x2d waiting on condition
> >>> [c2e7d000..c2e7dd98]
> >>>        at java.lang.Thread.sleep(Native Method)
> >>>        at org.apache.catalina.core.ContainerBase
> >>> $ContainerBackgroundProcessor.run(ContainerBase.java:1546)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-7" daemon prio=5 tid=0x087bf340 nid=0x1c in Object.wait()
> >>> [c42dd000..c42ddd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6585648> (a java.util.TaskQueue)
> >>>        at java.util.TimerThread.mainLoop(Timer.java:429)
> >>>        - locked <0xc6585648> (a java.util.TaskQueue)
> >>>        at java.util.TimerThread.run(Timer.java:382)
> >>>
> >>> "RMI LeaseChecker" daemon prio=5 tid=0x087521d8 nid=0x19 waiting
> >>> on condition [c43c9000..c43c9d98]
> >>>        at java.lang.Thread.sleep(Native Method)
> >>>        at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:
> >>> 307)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "RMI RenewClean-[172.21.35.85:9999]" daemon prio=5 tid=0x08410790
> >>> nid=0x17 in Object.wait() [c444d000..c444dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6509f90> (a java.lang.ref.ReferenceQueue
> >>> $Lock)
> >>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:
> >>> 111)
> >>>        - locked <0xc6509f90> (a java.lang.ref.ReferenceQueue$Lock)
> >>>        at sun.rmi.transport.DGCClient$EndpointEntry
> >>> $RenewCleanThread.run(DGCClient.java:500)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "GC Daemon" daemon prio=2 tid=0x0876f2c0 nid=0x15 in Object.wait
> >>> () [c44f5000..c44f5d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc6506288> (a sun.misc.GC$LatencyLock)
> >>>        at sun.misc.GC$Daemon.run(GC.java:100)
> >>>        - locked <0xc6506288> (a sun.misc.GC$LatencyLock)
> >>>
> >>> "RMI Reaper" prio=5 tid=0x08415a18 nid=0x14 in Object.wait()
> >>> [c4537000..c4537d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc63fac28> (a java.lang.ref.ReferenceQueue
> >>> $Lock)
> >>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:
> >>> 111)
> >>>        - locked <0xc63fac28> (a java.lang.ref.ReferenceQueue$Lock)
> >>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:
> >>> 127)
> >>>        at sun.rmi.transport.ObjectTable$Reaper.run
> >>> (ObjectTable.java:330)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "RMI TCP Accept-9999" daemon prio=5 tid=0x0849d668 nid=0x13
> >>> runnable [c4579000..c4579d98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc6506338> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at java.net.ServerSocket.accept(ServerSocket.java:419)
> >>>        at sun.rmi.transport.tcp.TCPTransport.run
> >>> (TCPTransport.java:334)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Socket Server: tcp://syisolx86vm1:4242" daemon prio=5
> >>> tid=0x084126f0 nid=0x12 runnable [c45bb000..c45bbd98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc6502430> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at java.net.ServerSocket.accept(ServerSocket.java:419)
> >>>        at org.activeio.net.SocketStreamChannelServer.accept
> >>> (SocketStreamChannelServer.java:66)
> >>>        at org.activeio.net.SocketSyncChannelServer.accept
> >>> (SocketSyncChannelServer.java:48)
> >>>        at org.activeio.adapter.SyncToAsyncChannelServer.run
> >>> (SyncToAsyncChannelServer.java:140)
> >>>        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor
> >>> $Worker.run(Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "LoginService login modules monitor" daemon prio=5 tid=0x08756c30
> >>> nid=0x11 in Object.wait() [c45fd000..c45fdd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc64fd090> (a
> >>> EDU.oswego.cs.dl.util.concurrent.ClockDaemon)
> >>>        at EDU.oswego.cs.dl.util.concurrent.ClockDaemon.nextTask
> >>> (Unknown Source)
> >>>        - locked <0xc64fd090> (a
> >>> EDU.oswego.cs.dl.util.concurrent.ClockDaemon)
> >>>        at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run
> >>> (Unknown Source)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "service.EJB@11369856" daemon prio=5 tid=0x0838ab68 nid=0x10
> >>> runnable [c485d000..c485dd98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc64b2910> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at java.net.ServerSocket.accept(ServerSocket.java:419)
> >>>        at org.activeio.xnet.ServiceDaemon$SocketListener.run
> >>> (ServiceDaemon.java:146)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "FlushManager" daemon prio=5 tid=0x08387790 nid=0xf waiting on
> >>> condition [c4951000..c4951d98]
> >>>        at java.lang.Thread.sleep(Native Method)
> >>>        at org.objectweb.howl.log.LogBufferManager$FlushManager.run
> >>> (LogBufferManager.java:1232)
> >>>
> >>> "LogFileManager.EventManager" daemon prio=5 tid=0x0871bed8
> >>> nid=0xe in Object.wait() [c4993000..c4993d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc649aee0> (a java.lang.Object)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.objectweb.howl.log.LogFileManager$EventManager.run
> >>> (LogFileManager.java:1220)
> >>>        - locked <0xc649aee0> (a java.lang.Object)
> >>>
> >>> "RMI TCP Accept-1099" daemon prio=5 tid=0x085b4cf8 nid=0xd
> >>> runnable [c4f79000..c4f79d98]
> >>>        at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> >>>        - locked <0xc63ef478> (a java.net.PlainSocketImpl)
> >>>        at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> >>>        at java.net.ServerSocket.accept(ServerSocket.java:419)
> >>>        at sun.rmi.transport.tcp.TCPTransport.run
> >>> (TCPTransport.java:334)
> >>>        at java.lang.Thread.run(Thread.java:534)
> >>>
> >>> "Thread-4" daemon prio=5 tid=0x085b4ae0 nid=0xc in Object.wait()
> >>> [c530b000..c530bd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc63ef550> (a java.util.TaskQueue)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at java.util.TimerThread.mainLoop(Timer.java:403)
> >>>        - locked <0xc63ef550> (a java.util.TaskQueue)
> >>>        at java.util.TimerThread.run(Timer.java:382)
> >>>
> >>> "Thread-2" daemon prio=5 tid=0x0841ac40 nid=0xb in Object.wait()
> >>> [c52c9000..c52c9d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc63b65f8> (a java.util.TaskQueue)
> >>>        at java.util.TimerThread.mainLoop(Timer.java:429)
> >>>        - locked <0xc63b65f8> (a java.util.TaskQueue)
> >>>        at java.util.TimerThread.run(Timer.java:382)
> >>>
> >>> "Thread-0" prio=5 tid=0x08406458 nid=0x9 in Object.wait()
> >>> [c534d000..c534dd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc63a73b0> (a java.util.TaskQueue)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at java.util.TimerThread.mainLoop(Timer.java:403)
> >>>        - locked <0xc63a73b0> (a java.util.TaskQueue)
> >>>        at java.util.TimerThread.run(Timer.java:382)
> >>>
> >>> "Signal Dispatcher" daemon prio=10 tid=0x08112358 nid=0x6 waiting
> >>> on condition [0..d017bd58]
> >>>
> >>> "Finalizer" daemon prio=8 tid=0x081046f0 nid=0x4 in Object.wait()
> >>> [d07ed000..d07edd98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc62ef960> (a java.lang.ref.ReferenceQueue
> >>> $Lock)
> >>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:
> >>> 111)
> >>>        - locked <0xc62ef960> (a java.lang.ref.ReferenceQueue$Lock)
> >>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:
> >>> 127)
> >>>        at java.lang.ref.Finalizer$FinalizerThread.run
> >>> (Finalizer.java:159)
> >>>
> >>> "Reference Handler" daemon prio=10 tid=0x08103cd8 nid=0x3 in
> >>> Object.wait() [d0848000..d0848d98]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc62ef9c8> (a java.lang.ref.Reference$Lock)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at java.lang.ref.Reference$ReferenceHandler.run
> >>> (Reference.java:115)
> >>>        - locked <0xc62ef9c8> (a java.lang.ref.Reference$Lock)
> >>>
> >>> "main" prio=5 tid=0x08072e90 nid=0x1 in Object.wait()
> >>> [8046000..8047030]
> >>>        at java.lang.Object.wait(Native Method)
> >>>        - waiting on <0xc630aa28> (a
> >>> org.apache.geronimo.kernel.basic.BasicKernel)
> >>>        at java.lang.Object.wait(Object.java:429)
> >>>        at org.apache.geronimo.system.main.Daemon.doStartup
> >>> (Daemon.java:321)
> >>>        - locked <0xc630aa28> (a
> >>> org.apache.geronimo.kernel.basic.BasicKernel)
> >>>        at org.apache.geronimo.system.main.Daemon.<init>
> >>> (Daemon.java:74)
> >>>        at org.apache.geronimo.system.main.Daemon.main(Daemon.java:
> >>> 367)
> >>>
> >>> "VM Thread" prio=5 tid=0x08102ec0 nid=0x2 runnable
> >>>
> >>> "VM Periodic Task Thread" prio=10 tid=0x08115bd0 nid=0x8 waiting
> >>> on condition
> >>> "Suspend Checker Thread" prio=10 tid=0x08111a20 nid=0x5 runnable
> >>>
> >>
> >>
> >
>
>


--
Regards,
Hiram

Mime
View raw message