tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dante Bell <DantePasqu...@cocoanet.us>
Subject Re: Urgent: Tomcat 6.0.20 on Solaris 10 Reaches max threads and is non responsive
Date Fri, 05 Aug 2011 16:10:16 GMT
This is probably a really dumb question, but say they implement
load-balanced Tomcat on 2 nodes for example. Would that then allow for
greater than 20 STMs for the servlets?

On 08/05/2011 12:00 PM, Mark Thomas wrote:
> On 05/08/2011 16:56, Dante Bell wrote:
>> Thanks!
>>
>> Like I said, I'm an OS/HW guy, never looked at java b4!
>>
>> They are saying that the load test has 20 'connections' so I'm guessing
>> that's the 20 STMs.
>>
>> Now, is this a fixable thing within the Java stack? Or is it an
>> application limitation?
> It looks to be hard-coded within Tomcat. I don't see a way to change
> that limit without building Tomcat from source.
>
> The other option is re-write the STM Servlet(s) as non-STM.
>
> Mark
>
>> Danté
>>
>> On 08/05/2011 11:12 AM, Mark Thomas wrote:
>>> On 05/08/2011 15:34, Dante Bell wrote:
>>>> Hi,
>>>>
>>>> I'm running out of ideas on what to try for this customer. Their load
>>>> tests show that Tomcat is getting to a point where it no longer services
>>>> requests.
>>> Let me guess. It is fine for low loads but as soon as the load goes
>>> above a certain number (maybe around 20 concurrent users?) then
>>> everything starts going wrong?
>>>
>>>> Thread dumps show most threads in a wait state, but some are
>>>> runnable. I'm suspecting GC is the issue,
>>> On what basis? The thread dump clearly shows a different problem.
>>>
>>>> but in analyzing the logs it
>>>> shows the longest pause was 15 seconds, but that only happened once, and
>>>> most of the stats look OK to me.
>>> That would suggest that it isn't GC then wouldn't it.
>>>
>>>> Details can be found on my blog: http://wp.me/plPvN-ai
>>> All the information needed to diagnose this issue is in the thead dump.
>>> If you take a look at the first thread that is blocked:
>>> "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in
>>> Object.wait() [0x2cd2f000..0x2cd2f870]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:485)
>>> 	at
>>> org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854)
>>> 	- locked <0x63a26708> (a java.util.Stack)
>>> 	at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)
>>> 	at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>
>>> And Bingo! we have found the problem.
>>>
>>> Now that stack trace might not ring alarm bells with someone unfamiliar
>>> with the Tomcat code base but if Tomcat is unresponsive, understanding
>>> why *any* thread is blocked would be a good place to start. If you look
>>> at line 854 and the surrounding code for StandardWrapper you will see
>>> that this is part of the Servlet allocation process. You should then
>>> realise that line 854 is part of Servlet allocation for Servlets that
>>> implement the SingleThreadModel (and now the alarm bells should be
>>> ringing loud and clear).
>>>
>>> Tomcat allocates a maximum of 20 instances of any STM servlet. Your
>>> client has an application that uses STM and requires many more than 20
>>> concurrent instances. Hence most requests are sat waiting for a STM
>>> instance to be released.
>>>
>>> Since that means there must be 20 instances of an STM Servlet already
>>> allocated, it is simple enough to grep the thread dump to find out which
>>> one. The winner is:
>>> com.motorola.nsm.common.ui.servlet.ValidateServlet
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message