tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pieter Temmerman <ptemmerman....@sadiel.es>
Subject Re: Thread dump analysis
Date Wed, 28 Jan 2009 12:45:41 GMT
I really appreciate your input Leon.

On Wed, 2009-01-28 at 11:07 +0100, Leon Rosenberg wrote:

> > "RMI TCP Connection(42)-173.x.x.x" - Thread t@112
> >   java.lang.Thread.State: RUNNABLE
> 
> reading from socket, usually not a problem.
> 
I thought so. Thanks.

> > "http-8081-35" - Thread t@71
> >   java.lang.Thread.State: RUNNABLE
> >        at java.lang.String.equals(String.java:1018)
> >        at
> > com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:554)
> >        at
> > org.codehaus.xfire.soap.handler.ReadHeadersHandler.invoke(ReadHeadersHandler.java:44)

> This one actually looks like a problem, was the server under load as
> you created the dump, or did you removed the load first?
No, the server was not under load at all. And I took various thread
dumps at random intervals, and they thread was still there (same thread
number). Maybe there is an infinite loop, but I don't know why it works
for a random amount of time, and then freezes.

> If there were no external requests to the server, you most probably
> have an infinite loop somewhere (how stable is xfire anyway?) or you
> have a really really really large xml file to parse. 
That's a good question. I have no experience with XFire. Would there be
a way to know which XML file it's parsing?

> 
> > "http-8081-31" - Thread t@67
> >   java.lang.Thread.State: RUNNABLE
> >        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl
> > $PrologDriver.next(XMLDocumentScannerImpl.java:930)
> >        at
> > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
> >        at
> > com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
> >        at
> > com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:548)
> >        at
> > org.codehaus.xfire.soap.handler.ReadHeadersHandler.invoke(ReadHeadersHandler.java:44)
> >        at
> > org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
> >        at
> > org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
> >        at
> > org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
> >        at
> > org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
> >        at
> > org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
> >        at
> > org.codehaus.xfire.transport.http.XFireServlet.doPost(XFireServlet.java:116)
> >
> 
> note that the XFire path is almost identical.
> Personally I neved managed to catch healthy code in something like
> String.equals(), but it may be possible, even imho unlikeable.

> The dump looks healthy, but is it the dump from the tomcat 5.5 or tomcat 6?
> Because if tomcat 6 is the problem and the dump is from tomcat 5 its useless :-)
> 
This was the heap dump from Tomcat 6.

> you need more research :-) For now it looks like your first tomcat is
> waiting for the second tomcat which just can't handle the requests as
> fast as its expected. You may want to check configuration options of
> both (threads in the connector etc).
If the threadMax would be too low in the connector, wouldn't the
"freeze" be over once there are free connections? And also, how can a
small threadMax make a thread hang? For example the one that is trying
to read an XML file.

> Also the heap dump of the second tomcat would be useful (if this one
> is from first).
> And a thread dump shortly after the traffic has gone (for example kill
> first tomcat, wait 1 minute, make thread dump).
> 
Did this, see the output (I removed the RMI threads):

"TP-Monitor" - Thread t@28
   java.lang.Thread.State: TIMED_WAITING on org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable@498364
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"TP-Processor4" - Thread t@27
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked java.net.SocksSocketImpl@134af1
	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:306)
	at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:660)
	at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:870)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"TP-Processor3" - Thread t@26
   java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@514577
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"TP-Processor2" - Thread t@25
   java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@1a79b48
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"TP-Processor1" - Thread t@24
   java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@c3a53b
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"http-8081-Acceptor-0" - Thread t@23
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked java.net.SocksSocketImpl@9d478b
	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@22
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"Monitor Runner" - Thread t@21
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at com.icesoft.util.MonitorRunner$1.run(MonitorRunner.java:20)

   Locked ownable synchronizers:
	- None

"pool-1-thread-1" - Thread t@20
   java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@94c5e7
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"Store org.hibernate.cache.StandardQueryCache Spool Thread" - Thread t@18
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:563)
	at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:64)
	at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1047)

   Locked ownable synchronizers:
	- None

"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" - Thread t@17
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:563)
	at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:64)
	at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1047)

   Locked ownable synchronizers:
	- None

"HSQLDB Timer @55d1b4" - Thread t@16
   java.lang.Thread.State: TIMED_WAITING on org.hsqldb.lib.HsqlTimer$TaskQueue@14cff90
	at java.lang.Object.wait(Native Method)
	at org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)
	at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
	at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
	- None

"Timer-0" - Thread t@15
   java.lang.Thread.State: WAITING on java.util.TaskQueue@1e6661
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at java.util.TimerThread.mainLoop(Timer.java:483)
	at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
	- None

"Session Monitor" - Thread t@14
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Listener$1.run(SessionDispatcher.java:208)

   Locked ownable synchronizers:
	- None

"Store userCache Spool Thread" - Thread t@13
   java.lang.Thread.State: TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:573)
	at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
	at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1057)

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" - Thread t@4
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" - Thread t@3
   java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@1d36f77
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

   Locked ownable synchronizers:
	- None

"Reference Handler" - Thread t@2
   java.lang.Thread.State: WAITING on java.lang.ref.Reference$Lock@3b1799
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

   Locked ownable synchronizers:
	- None

"main" - Thread t@1
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked java.net.SocksSocketImpl@735aa1
	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
	at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

   Locked ownable synchronizers:
	- None




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message