Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Message-ID: <4E3C0877.1040109@apache.org>
Date: Fri, 05 Aug 2011 16:12:55 +0100
From: Mark Thomas <markt@apache.org>
User-Agent: Mozilla/5.0 (Windows NT 5.2; WOW64;
 rv:5.0) Gecko/20110624 Thunderbird/5.0
MIME-Version: 1.0
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Urgent: Tomcat 6.0.20 on Solaris 10 Reaches max threads and is
 non responsive
References: <4E3BFF5F.2070605@cocoanet.us>
In-Reply-To: <4E3BFF5F.2070605@cocoanet.us>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/08/2011 15:34, Dante Bell wrote:
> Hi,
> 
> I'm running out of ideas on what to try for this customer. Their load
> tests show that Tomcat is getting to a point where it no longer services
> requests.

Let me guess. It is fine for low loads but as soon as the load goes
above a certain number (maybe around 20 concurrent users?) then
everything starts going wrong?

> Thread dumps show most threads in a wait state, but some are
> runnable. I'm suspecting GC is the issue,

On what basis? The thread dump clearly shows a different problem.

> but in analyzing the logs it
> shows the longest pause was 15 seconds, but that only happened once, and
> most of the stats look OK to me.

That would suggest that it isn't GC then wouldn't it.

> Details can be found on my blog: http://wp.me/plPvN-ai

All the information needed to diagnose this issue is in the thead dump.
If you take a look at the first thread that is blocked:
"TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in
Object.wait() [0x2cd2f000..0x2cd2f870]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at
org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854)
	- locked <0x63a26708> (a java.util.Stack)
	at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)
	at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

And Bingo! we have found the problem.

Now that stack trace might not ring alarm bells with someone unfamiliar
with the Tomcat code base but if Tomcat is unresponsive, understanding
why *any* thread is blocked would be a good place to start. If you look
at line 854 and the surrounding code for StandardWrapper you will see
that this is part of the Servlet allocation process. You should then
realise that line 854 is part of Servlet allocation for Servlets that
implement the SingleThreadModel (and now the alarm bells should be
ringing loud and clear).

Tomcat allocates a maximum of 20 instances of any STM servlet. Your
client has an application that uses STM and requires many more than 20
concurrent instances. Hence most requests are sat waiting for a STM
instance to be released.

Since that means there must be 20 instances of an STM Servlet already
allocated, it is simple enough to grep the thread dump to find out which
one. The winner is:
com.motorola.nsm.common.ui.servlet.ValidateServlet

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org