tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From balvindar dhaliwal <balvind...@hotmail.com>
Subject RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
Date Fri, 14 Sep 2012 12:09:15 GMT

Hi,
The issue seems similar to:
http://www.tomcatexpert.com/ask-the-experts/tomcat-hangs-production-and-doesn%E2%80%99t-respond-new-http-requests

 

https://issues.apache.org/bugzilla/show_bug.cgi?id=53173
Regards,Bal.
From: balvindard@hotmail.com
To: users@tomcat.apache.org
Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
Date: Tue, 11 Sep 2012 15:04:45 +0000





Hi,

I have pasted the first thread dump, and attached 3 other thread dumps.
Thread dump1:-------------------
2012-08-10 14:40:52
Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):

"RMI TCP Connection(23136)-149.184.142.160" - Thread t@37810
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@427137
	at java.io.FilterInputStream.read(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@17119c

"JMX server connection timeout 37809" - Thread t@37809
   java.lang.Thread.State: TIMED_WAITING on [I@378cf5
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"JMX server connection timeout 37808" - Thread t@37808
   java.lang.Thread.State: TIMED_WAITING on [I@1075926
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"RMI TCP Connection(23129)-149.177.35.85" - Thread t@37806
   java.lang.Thread.State: RUNNABLE
	at sun.management.ThreadImpl.dumpThreads0(Native Method)
	at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)
	at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
	at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
	at javax.management.StandardMBean.invoke(Unknown Source)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
	at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor5299.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@d0793c

"RMI TCP Connection(23134)-149.184.142.160" - Thread t@37803
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@754035
	at java.io.FilterInputStream.read(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1ca727a

"RMI TCP Connection(23130)-149.184.142.160" - Thread t@37799
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@d3f8d
	at java.io.FilterInputStream.read(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1c75893

"RMI TCP Connection(23115)-149.184.142.160" - Thread t@37795
   java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
	at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
	at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@88281d

"RMI TCP Connection(23135)-149.184.142.160" - Thread t@37787
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@9d3697
	at java.io.FilterInputStream.read(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@16af9f9

"RMI TCP Connection(23062)-149.184.142.160" - Thread t@37756
   java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
	at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
	at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@415da0

"http-bio-8443-exec-13182" - Thread t@37623
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@169bcbd
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@e61d04
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked org.apache.tomcat.util.net.SocketWrapper@17b14f2
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@19c4cd7

"http-bio-8443-exec-13181" - Thread t@37619
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-exec-13180" - Thread t@37618
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-exec-13179" - Thread t@37615
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-exec-13176" - Thread t@37584
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@9d2f3
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@f8902a
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked org.apache.tomcat.util.net.SocketWrapper@c0d197
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1004e70

"http-bio-8443-exec-13175" - Thread t@37583
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-exec-13174" - Thread t@37582
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-exec-13172" - Thread t@37580
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
	at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-exec-13157" - Thread t@37493
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@11db953
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@7200ec
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked org.apache.tomcat.util.net.SocketWrapper@9aa1e0
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1f9b5f5

"http-bio-8443-exec-13153" - Thread t@37489
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@118f940
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@93e184
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked org.apache.tomcat.util.net.SocketWrapper@bae208
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@14d8afa

"JMX server connection timeout 20278" - Thread t@20278
   java.lang.Thread.State: TIMED_WAITING on [I@172a3d4
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"JMX server connection timeout 20277" - Thread t@20277
   java.lang.Thread.State: TIMED_WAITING on [I@161b40a
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-8211461)" - Thread t@14697
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@15c19ae
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@199ff9e
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@15cf15b
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-7262243)" - Thread t@6265
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@2e0c34
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1233759
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@106d785
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-34701031)" - Thread t@4657
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@2f626a
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@e655a9
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@2000c
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-10500556)" - Thread t@4580
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@1514a33
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1de6f76
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@67b838
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-10479806)" - Thread t@3911
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@16f6ae4
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2d6dd
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@1185eab
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-7530293)" - Thread t@3746
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@ef6481
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@af7fcb
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@1eef61c
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-7208323)" - Thread t@3705
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@6966e3
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1e1e638
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@1d1a81f
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS Connections Pinger" - Thread t@3704
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"Timer-1" - Thread t@3630
   java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1f564da
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-7291923)" - Thread t@677
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@174b00f
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2b3ca
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@19a318
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-7097900)" - Thread t@380
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@1229e8f
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d7cf47
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@1697394
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-10324836)" - Thread t@157
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@22fe5f
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@16a4569
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@1f3009b
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"Timer-0" - Thread t@155
   java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1b0504c
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-34276374)" - Thread t@97
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@acfea9
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@1832a6b
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@55e5f0
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS TCPLink Reader (Server-7093946)" - Thread t@91
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked java.lang.Object@16af6ea
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	- locked com.sun.net.ssl.internal.ssl.AppInputStream@17891e4
	at java.io.BufferedInputStream.fill(Unknown Source)
	at java.io.BufferedInputStream.read(Unknown Source)
	- locked java.io.BufferedInputStream@b2270e
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
	at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
	at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)

   Locked ownable synchronizers:
	- None

"TIBCO EMS Connections Pinger" - Thread t@90
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"Java2D Disposer" - Thread t@51
   java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@67dfae
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
	at sun.java2d.Disposer.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-AsyncTimeout" - Thread t@32
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:141)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"http-bio-8443-Acceptor-0" - Thread t@31
   java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.LimitLatch$Sync@16dd685
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
	at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:99)
	at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:581)
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:204)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@30
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1369)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"GC Daemon" - Thread t@21
   java.lang.Thread.State: TIMED_WAITING on sun.misc.GC$LatencyLock@18e5957
	at java.lang.Object.wait(Native Method)
	at sun.misc.GC$Daemon.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"RMI Scheduler(0)" - Thread t@14
   java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@186b6cc
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
	at java.util.concurrent.DelayQueue.take(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"RMI TCP Accept-7360" - Thread t@11
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(Unknown Source)
	- locked java.net.SocksSocketImpl@1ad9253
	at java.net.ServerSocket.implAccept(Unknown Source)
	at java.net.ServerSocket.accept(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"RMI TCP Accept-0" - Thread t@10
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(Unknown Source)
	- locked java.net.SocksSocketImpl@61c8a1
	at java.net.ServerSocket.implAccept(Unknown Source)
	at java.net.ServerSocket.accept(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" - Thread t@4
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" - Thread t@3
   java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@14b402d
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
	at java.lang.ref.ReferenceQueue.remove(Unknown Source)
	at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"Reference Handler" - Thread t@2
   java.lang.Thread.State: WAITING on java.lang.ref.Reference$Lock@16269cb
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

   Locked ownable synchronizers:
	- None

"main" - Thread t@1
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(Unknown Source)
	- locked java.net.SocksSocketImpl@94237f
	at java.net.ServerSocket.implAccept(Unknown Source)
	at java.net.ServerSocket.accept(Unknown Source)
	at org.apache.catalina.core.StandardServer.await(StandardServer.java:447)
	at org.apache.catalina.startup.Catalina.await(Catalina.java:709)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:654)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)

   Locked ownable synchronizers:
	- None


Also, we are using Default JPA Connection Pooling? Should we use Tomcat JDBC Connection Pooling?
There nothing additional configured in context.xml.
I see UserDatabaseRealm in server.xml, is this a default setting?
What connector is recommended for SSL Connections on Solaris 10?

Regards,Bal.


Balvindar,

On 9/3/12 9:12 AM, balvindar dhaliwal wrote:
We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web
application. Here are the software versions and hardware
infrastructure details. Apache Tomact # 7.0.19 Java run time  #
1.6.0_12


If possible, it would be best to upgrade to the latest version of both
of these packages: Java 1.6.0_36 and Tomcat 7.0.30. There are several
important security fixes in both of those (relative to where you are now).

We have noticed after about 2 months, the web requests stop being 
processed. We had observed this issue twice in the last 5 months.
We are using HTTP/1.1 Connector ( BIO ). The heap and permgen
memory seem fine. There is no out of memory errors in the logs. We
have used jvisualvm to monitor heap, permgen, threads and thread
dump, AND THEY ALL LOOK OK.   The thread dump looks fine and there
are no deadlocks/stuck threads observed.


Your thread dump attachment was stripped from the list. Can you
copy/paste it into a follow post?

Please let me know if you need more information.


It's hard to guess without the thread dump, but my initial reaction is
that it's connection-pool exhaustion, but you said that your
connection pool seems healthy.

I see from your server.xml that you are using UserDatabaseRealm: is
that actually in use in production? It's probably an inappropriate
component to use for anything but the most trivial configuration (like
maybe the Tomcat manager app). Are you using any kind of <Realm> or
<Resource> specifically configured in in your webapp's context.xml?
That would also be good to know.

If you are using JDBCRealm, you may have hit an edge case that causes
some kind of deadlock (but again, you say there are no deadlocks).

There were some fixes to the NIO and APR connectors recently (and your
version of Tomcat is now more than a year out-of-date) that fixed
apparent deadlocks but I'm not sure if those extended into the BIO
realm. I can't see anything in the changelog that would indicate a fix
to the BIO connector that would affect you.

Again, it's very difficult to speculate without some more data, so
sending that thread dump (especially if you can give us several thread
dumps over a few minutes) will definitely help.

> From: balvindard@hotmail.com
> To: users@tomcat.apache.org
> Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
> Date: Mon, 3 Sep 2012 13:47:50 +0000
> 
> 
> 
> Hi,
> Q1:server.xml :------------<?xml version='1.0' encoding='utf-8'?><Server port="8009" shutdown="xxxxx">  <Listener className="org.apache.catalina.security.SecurityListener" />  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />  <Listener className="org.apache.catalina.core.JasperListener" />  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />  <GlobalNamingResources>    <Resource name="UserDatabase" auth="Container"              type="org.apache.catalina.UserDatabase"              description="User database that can be updated and saved"              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"              pathname="conf/tomcat-users.xml" />  </GlobalNamingResources>  <Service name="Catalina">    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"               maxThreads="150" scheme="https" secure="true"               clientAuth="false" sslProtocol="TLS"               ciphers="TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"               keystoreFile="conf/xxxxxx-yy-zzzz.jks" keystorePass="xxxx" />    <Engine name="Catalina" defaultHost="localhost">      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"             digest="SHA" resourceName="UserDatabase"/>      <Host name="localhost" appBase="webapps"            unpackWARs="true" autoDeploy="true"            xmlValidation="false" xmlNamespaceAware="false">
>         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
>       </Host>    </Engine>  </Service></Server>
> Q2:We are using HTTPS.
> Regards,Bal.
> > Date: Mon, 3 Sep 2012 15:35:31 +0200
> > From: aw@ice-sa.com
> > To: users@tomcat.apache.org
> > Subject: Re: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
> > 
> > balvindar dhaliwal wrote:
> > >  
> > > 
> > > 
> > > 
> > > Hi ,
> > > 
> > > 
> 
> > > We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application. 
> > > Here are the software versions and hardware infrastructure details.
> > > Apache Tomact # 7.0.19
> > > Java run time  # 1.6.0_12
> > > Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc SUNW,Sun-Fire-T200
> > > OS version # solaris 10.
> > >  
> > > Issue Description # 
> > > We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months. 
> > > We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK.   The thread dump looks fine and there are no deadlocks/stuck threads observed. 
> > > The localhost_access_log stops being updated. The connector port is still listening on SSL port 8443.
> > > After we restart Tomcat, the new requests are processing fine.
> > > Can you please advise other areas we should look into for investigation (any additional tracing).
> > > Should we upgrade to tomcat 7.0.29? What could cause the Connector to stop processing requests on the port?
> > > Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively reducing the heap. The fact that local access log is not being updated, suggests the request is not making to the Servlet container. We are using JPA Connection Pool, have not noticed any db connection issues. After the restart, everything works well.
> > > The system logs seems to be fine. 
> > > Please find the attached sample thread dump copy when this issue was occurred. 
> > >  
> > > Please let me know if you need more information. 
> > >  
> > 
> > Q1: Can you copy and paste the server.xml of Tomcat here ? (removing any private information).
> > Copy and paste in-line, the list strips most attachments.
> > 
> > Q2: Above, you mention that the HTTPS connector on port 8443 is still listening. What 
> > about the HTTP connector on port ??? (usually 80).
> > Or are all the requests HTTPS ?
> > 
> > 
> >    		 	   		
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> > 
>  		 	   		  
 		 	   		  

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message