You may want to check first which "keep alive" settings they have for the whole chain.
If one of them happened to be close to 15 minutes, that might be a strong hint.
Another tool is "netstat", to see exactly in which TCP state these persistent connections
find themselves in. (another bad sentence, just keeping in style here).
Dante Bell wrote:
> Hi Chris,
>
> I did indeed read and digest Mark's email and talked to the vendor about
> that issue. The stack trace on the old blog post is from the one Mark
> was helping out with (man, that was a really bad sentance!).
>
> This is a different issue :( I don't have a stack trace and I don't have
> access to the lab they are running these tests in. I've requested the
> stack traces when this happens, but haven't received those yet.
>
> Your question about 'crash' is valid and the explanation I received was
> that the load test application crashes. That's all I have at this time
> from them. I'm helping them from a dark, distant planet and only see the
> things they want me to see ;) Weirdly, it doesn't sound like TC is dead
> from what they are telling me, after 15 minutes it starts serving up db
> responses!
>
> Yes, they are using mod_jk.
>
>
>
> On 08/10/2011 12:55 PM, Christopher Schultz wrote:
>> Dante,
>>
>> On 8/10/2011 11:57 AM, Dante Bell wrote:
>>> We are seeing that after an application crash (customized load
>>> tester with minimal error handling so it crashes often)
>> When you say "crash", do you mean you get a stack trace in the logs and
>> Tomcat stays up, or do you mean that you bring-down the JVM? If you
>> bring-down the JVM, what is the error that is occurring (check hs_*.txt
>> files laying around in the working directory for that)?
>>
>>> that TC isn't releasing the connection for about 15 minutes.
>> If TC is truly dead, then it's not holding connections at all. That
>> would be the OS holding them.
>>
>> What makes you think they are not being "released"? What counts as
>> "released"?
>>
>>> I've reviewed some of the worker directives, but I'm really unsure as
>>> to which one or combination would shorten this interval
>>> significantly.
>> Does that mean you are using mod_jk/mod_proxy_ajp? Good to have that
>> kind of information.
>>
>>> The Apache server still serves up static content, which makes me
>>> think that there isn't anything at the OS or Apache layer that is
>>> causing the connection to hang around (granted, this isn't an
>>> absolute and we are investigating these 2 components also).
>> So you're using Apache httpd, too. Also good to know.
>>
>>> We've done some minor TCP/IP tuning in the Solaris stack, and that
>>> has helped with other issues regarding heavy loads.
>> On Solaris.
>>
>>> If TC is the culprit, would we need to be setting the advanced
>>> connector directives such as:
>>> |recovery_options |4: close the connection to Tomcat, if we
>>> detect an error when writing back the answer to the client (browser)
>> That depends upon what the errors actually are. Care to tell us about
>> them?
>>
>>> PS. Configs can be found at: http://bit.ly/pFIzO0
>> Sigh. You should look into "template" workers.
>>
>> Apache httpd MaxClients setting default is 256. <Connector> MaxThreads
>> is set to 750, so Tomcat should have almost 3 times more than you need.
>> Where do you see 750 stuck threads?
>>
>> I looked at your thread dump. You clearly have not read Mark's previous
>> response on this list where he told you exactly what was happening: your
>> webapp is killing itself with these SingleThreadModel servlets. This is
>> not thread starvation due to configuration, this is thread starvation
>> due to a poorly-implemented web application.
>>
>>> Apache:* Apache HTTP Server Version 2.2 -- prefork with mpm *Tomcat:*
>>> 6.0.20 *JK Connector:* Same as whatever is bundled in with Apache 2.2
>>> (from customer) *Solaris* Solaris 10 10/09 s10s_u8wos_08a SPARC
>> Aah, here's all the configuration information. Description then context.
>> Not the best term paper I've ever read. :(
>>
>> I think you mean "prefork MPM". Apache httpd does not bundle mod_jk.
>> Check your version.
>>
>> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
|