tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dante Bell <>
Subject Re: Urgent: Tomcat 6.0.20 on Solaris 10 Reaches max threads and is non responsive
Date Sat, 06 Aug 2011 02:57:19 GMT
Hi Guys,

I hate to pick your brains on this as the customer should know how to do
this, but they tasked me to find out ;(

Is there any API or other method they can code that will give them an
indication of nearing this threshold? I know it's a crappy solution, but
I thought I'd ask anyway :) AFAIK, from the OS side there's really
nothing I can think of that would help.

Thanks for all your help, it's greatly appreciated by myself and the
entire team,

On 08/05/2011 12:14 PM, Mark Thomas wrote:
> On 05/08/2011 17:10, Dante Bell wrote:
>> This is probably a really dumb question, but say they implement
>> load-balanced Tomcat on 2 nodes for example. Would that then allow for
>> greater than 20 STMs for the servlets?
> It will allow them up to 20 concurrent requests per STM Servlet per
> Tomcat instance. What that means for actual users is difficult to judge
> since a single request may be routed through multiple servlets
> (including the same servlet several times). It will certainly increase
> capacity. From what to what is hard impossible to tell without knowing
> the application code.
> Mark
>> On 08/05/2011 12:00 PM, Mark Thomas wrote:
>>> On 05/08/2011 16:56, Dante Bell wrote:
>>>> Thanks!
>>>> Like I said, I'm an OS/HW guy, never looked at java b4!
>>>> They are saying that the load test has 20 'connections' so I'm guessing
>>>> that's the 20 STMs.
>>>> Now, is this a fixable thing within the Java stack? Or is it an
>>>> application limitation?
>>> It looks to be hard-coded within Tomcat. I don't see a way to change
>>> that limit without building Tomcat from source.
>>> The other option is re-write the STM Servlet(s) as non-STM.
>>> Mark
>>>> Danté
>>>> On 08/05/2011 11:12 AM, Mark Thomas wrote:
>>>>> On 05/08/2011 15:34, Dante Bell wrote:
>>>>>> Hi,
>>>>>> I'm running out of ideas on what to try for this customer. Their
>>>>>> tests show that Tomcat is getting to a point where it no longer services
>>>>>> requests.
>>>>> Let me guess. It is fine for low loads but as soon as the load goes
>>>>> above a certain number (maybe around 20 concurrent users?) then
>>>>> everything starts going wrong?
>>>>>> Thread dumps show most threads in a wait state, but some are
>>>>>> runnable. I'm suspecting GC is the issue,
>>>>> On what basis? The thread dump clearly shows a different problem.
>>>>>> but in analyzing the logs it
>>>>>> shows the longest pause was 15 seconds, but that only happened once,
>>>>>> most of the stats look OK to me.
>>>>> That would suggest that it isn't GC then wouldn't it.
>>>>>> Details can be found on my blog:
>>>>> All the information needed to diagnose this issue is in the thead dump.
>>>>> If you take a look at the first thread that is blocked:
>>>>> "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in
>>>>> Object.wait() [0x2cd2f000..0x2cd2f870]
>>>>>    java.lang.Thread.State: WAITING (on object monitor)
>>>>> 	at java.lang.Object.wait(Native Method)
>>>>> 	at java.lang.Object.wait(
>>>>> 	at
>>>>> org.apache.catalina.core.StandardWrapper.allocate(
>>>>> 	- locked <0x63a26708> (a java.util.Stack)
>>>>> 	at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(
>>>>> 	at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(
>>>>> And Bingo! we have found the problem.
>>>>> Now that stack trace might not ring alarm bells with someone unfamiliar
>>>>> with the Tomcat code base but if Tomcat is unresponsive, understanding
>>>>> why *any* thread is blocked would be a good place to start. If you look
>>>>> at line 854 and the surrounding code for StandardWrapper you will see
>>>>> that this is part of the Servlet allocation process. You should then
>>>>> realise that line 854 is part of Servlet allocation for Servlets that
>>>>> implement the SingleThreadModel (and now the alarm bells should be
>>>>> ringing loud and clear).
>>>>> Tomcat allocates a maximum of 20 instances of any STM servlet. Your
>>>>> client has an application that uses STM and requires many more than 20
>>>>> concurrent instances. Hence most requests are sat waiting for a STM
>>>>> instance to be released.
>>>>> Since that means there must be 20 instances of an STM Servlet already
>>>>> allocated, it is simple enough to grep the thread dump to find out which
>>>>> one. The winner is:
>>>>> com.motorola.nsm.common.ui.servlet.ValidateServlet
>>>>> Mark
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail:
>>>>> For additional commands, e-mail:
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message