tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leon Rosenberg <rosenberg.l...@googlemail.com>
Subject Re: Thread dump analysis
Date Wed, 28 Jan 2009 10:07:56 GMT
On Wed, Jan 28, 2009 at 10:42 AM, Pieter Temmerman
<ptemmerman.ext@sadiel.es> wrote:
>
>
>>Have you found any java.lang.Thread.State: RUNNABLE threads? They are
>>usually more interesting if it comes to a high cpu :-)
>
> These are the RUNNABLE threads on Tomcat 6:
>
> "RMI TCP Connection(42)-173.x.x.x" - Thread t@112
>   java.lang.Thread.State: RUNNABLE

reading from socket, usually not a problem.

> "http-8081-35" - Thread t@71
>   java.lang.Thread.State: RUNNABLE
>        at java.lang.String.equals(String.java:1018)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:554)
>        at
> org.codehaus.xfire.soap.handler.ReadHeadersHandler.invoke(ReadHeadersHandler.java:44)
>        at
> org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
>        at
> org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
>        at
> org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
>        at
> org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
>        at
> org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
>        at
> org.codehaus.xfire.transport.http.XFireServlet.doPost(XFireServlet.java:116)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>        at

This one actually looks like a problem, was the server under load as
you created the dump, or did you removed the load first?
If there were no external requests to the server, you most probably
have an infinite loop somewhere (how stable is xfire anyway?) or you
have a really really really large xml file to parse. If there were
requests to the server as you took the dump, you might just happen to
catch this part of the execution. In this case you should remove the
traffic first and create a dump afterwards (of course only if the load
remains after the traffic is cut off)


> "http-8081-31" - Thread t@67
>   java.lang.Thread.State: RUNNABLE
>        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl
> $PrologDriver.next(XMLDocumentScannerImpl.java:930)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:548)
>        at
> org.codehaus.xfire.soap.handler.ReadHeadersHandler.invoke(ReadHeadersHandler.java:44)
>        at
> org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
>        at
> org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
>        at
> org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
>        at
> org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
>        at
> org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
>        at
> org.codehaus.xfire.transport.http.XFireServlet.doPost(XFireServlet.java:116)
>

note that the XFire path is almost identical.
Personally I neved managed to catch healthy code in something like
String.equals(), but it may be possible, even imho unlikeable.

>
> The Tomcat 5.5 webapp which connects to the webservice is not
> responding. I saw the following thread on Tomcat 5.5 which may be the
> connection to the webservice:
>
> The fact that it's reading from a socket, which I think it the
> connection to the webservice, may explain why the application is not
> doing anything.

Yep

>
>>Also, as David posted, what is the HEAP usage? it's usually at the end
>>of the dump.
>
> vdebian2:/usr/local/jdk1.6.0_07/bin# ./jmap 8100
> Attaching to process ID 8100, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 10.0-b23
>
> using thread-local object allocation.
> Parallel GC with 2 thread(s)
>
> Heap Configuration:
>   MinHeapFreeRatio = 40
>   MaxHeapFreeRatio = 70
>   MaxHeapSize      = 1073741824 (1024.0MB)
>   NewSize          = 1048576 (1.0MB)
>   MaxNewSize       = 4294901760 (4095.9375MB)
>   OldSize          = 4194304 (4.0MB)
>   NewRatio         = 8
>   SurvivorRatio    = 8
>   PermSize         = 16777216 (16.0MB)
>   MaxPermSize      = 268435456 (256.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
>   capacity = 58851328 (56.125MB)
>   used     = 18383664 (17.532028198242188MB)
>   free     = 40467664 (38.59297180175781MB)
>   31.237466722925948% used
> >From Space:
>   capacity = 327680 (0.3125MB)
>   used     = 131072 (0.125MB)
>   free     = 196608 (0.1875MB)
>   40.0% used
> To Space:
>   capacity = 393216 (0.375MB)
>   used     = 0 (0.0MB)
>   free     = 393216 (0.375MB)
>   0.0% used
> PS Old Generation
>   capacity = 477233152 (455.125MB)
>   used     = 86499376 (82.49223327636719MB)
>   free     = 390733776 (372.6327667236328MB)
>   18.125181714115286% used
> PS Perm Generation
>   capacity = 81395712 (77.625MB)
>   used     = 62345160 (59.45697784423828MB)
>   free     = 19050552 (18.16802215576172MB)
>   76.59514054008152% used


The dump looks healthy, but is it the dump from the tomcat 5.5 or tomcat 6?
Because if tomcat 6 is the problem and the dump is from tomcat 5 its useless :-)

>
> I'm quite sure that one of the RUNNING threads of Tomcat 6 is the
> problem, which will probably be related to the webservice.
> But I'm puzzled with wath the problem might be.

you need more research :-) For now it looks like your first tomcat is
waiting for the second tomcat which just can't handle the requests as
fast as its expected. You may want to check configuration options of
both (threads in the connector etc).
Also the heap dump of the second tomcat would be useful (if this one
is from first).
And a thread dump shortly after the traffic has gone (for example kill
first tomcat, wait 1 minute, make thread dump).

regards
Leon

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message