tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: Urgent: Tomcat 6.0.20 on Solaris 10 Reaches max threads and is non responsive
Date Fri, 05 Aug 2011 15:56:19 GMT
Pid wrote:
> On 05/08/2011 16:12, Mark Thomas wrote:
>> On 05/08/2011 15:34, Dante Bell wrote:
>>> Hi,
>>>
>>> I'm running out of ideas on what to try for this customer. Their load
>>> tests show that Tomcat is getting to a point where it no longer services
>>> requests.
>> Let me guess. It is fine for low loads but as soon as the load goes
>> above a certain number (maybe around 20 concurrent users?) then
>> everything starts going wrong?
>>
>>> Thread dumps show most threads in a wait state, but some are
>>> runnable. I'm suspecting GC is the issue,
>> On what basis? The thread dump clearly shows a different problem.
>>
>>> but in analyzing the logs it
>>> shows the longest pause was 15 seconds, but that only happened once, and
>>> most of the stats look OK to me.
>> That would suggest that it isn't GC then wouldn't it.
>>
>>> Details can be found on my blog: http://wp.me/plPvN-ai
>> All the information needed to diagnose this issue is in the thead dump.
>> If you take a look at the first thread that is blocked:
>> "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in
>> Object.wait() [0x2cd2f000..0x2cd2f870]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:485)
>> 	at
>> org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854)
>> 	- locked <0x63a26708> (a java.util.Stack)
>> 	at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)
>> 	at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>
>> And Bingo! we have found the problem.
>>
>> Now that stack trace might not ring alarm bells with someone unfamiliar
>> with the Tomcat code base but if Tomcat is unresponsive, understanding
>> why *any* thread is blocked would be a good place to start. If you look
>> at line 854 and the surrounding code for StandardWrapper you will see
>> that this is part of the Servlet allocation process. You should then
>> realise that line 854 is part of Servlet allocation for Servlets that
>> implement the SingleThreadModel (and now the alarm bells should be
>> ringing loud and clear).
>>
>> Tomcat allocates a maximum of 20 instances of any STM servlet. Your
>> client has an application that uses STM and requires many more than 20
>> concurrent instances. Hence most requests are sat waiting for a STM
>> instance to be released.
>>
>> Since that means there must be 20 instances of an STM Servlet already
>> allocated, it is simple enough to grep the thread dump to find out which
>> one. The winner is:
>> com.motorola.nsm.common.ui.servlet.ValidateServlet
> 
> "By Jove, Holmes!", exclaimed Dr Watson.
> 

Brilliant use of ze little grey cells !
(s) Hercule Poirot

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message