tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Chheda <chetan_chh...@yahoo.com>
Subject Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
Date Thu, 21 May 2009 18:08:35 GMT
I am following this thread with great interest. I have a similar issue as Vishwajit and have
resorted to adding the connectionTimeout to get rid of a large number of RUNNABLE threads.


But mod_jk does not like tomcat threads timing out and logs the message "increase the backend
idle connection timeout or
 the connection_pool_minsize" in the mod_jk logs

which leads me to believe that its apache thats not letting go of the threads in my case.



________________________________
From: Christopher Schultz <chris@christopherschultz.net>
To: Tomcat Users List <users@tomcat.apache.org>
Sent: Thursday, May 21, 2009 1:05:21 PM
Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no
activity

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vishwajit,

On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote:
> [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should
> not be an issue. The only reason why I thought that was an issue was
> that I was observing that the none of the RUNNABLE connections were
> not being used to serve new requests, only the WAITING ones were -
> and I do know for sure that the RUNNABLE threads were not servicing
> any existing requests as I was the only one using the system then.

It seems pretty clear that this is what your problem is. See if you can
follow the order of events described below:

1. Tomcat and Apache httpd are started. httpd makes one or more
  (persistent) AJP connections to Tomcat and holds them open (duh).
  Each connection from httpd->Tomcat puts a Java thread in the RUNNABLE
  state (though actually blocked on socket read, it's not "really"
  runnable)

2. Some requests are received by httpd and sent over the AJP connections
  to Tomcat (or not ... it really doesn't matter)

3. Time passes, your recycle_timeout (300s) or cache_timeout (600s)
  expires

4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully
  follows your instructions for closing the connections expired in #3
  above (note that Tomcat has no idea that the connection has been
  closed, and so those threads remain in the RUNNABLE state, not
  connected to anything, lost forever)

5. A new connection (or multiple new connections... not sure exactly
  how mod_jks connection expiration-and-reconnect logic is done)
  is made to Tomcat which allocates a new thread (or threads)
  which is/are in the RUNNABLE state

Rinse, repeat, your server chokes to death when it runs out of threads.

The above description accounts for your "loss" of 4 threads at a time:
your web browser requests the initial page followed by 3 other assets
(image, css, whatever). Each one of them hits step #4 above, causing a
new AJP connection to be created, with the old one still hanging around
on the Tomcat side just wasting a thread and memory.

By setting connectionTimeout on the AJP <Connector>, you are /doing what
you should have done in the first place, which is match mod_jk
cache_timeout with Connector connectionTimeout/. This allows the threads
on the Tomcat side to expire just like those on the httpd side. They
should expire at (virtually) the same time and everything works as expected.

This problem is compounded by your initial configuration which created
10 connections from httpd->Tomcat for every (prefork) httpd process,
resulting in 9 useless AJP connections for every httpd process. I
suspect that you were expiring 10 connections at a time instead of just
one, meaning that you were running out of threads 10 times faster than
you otherwise would.

Suggestions:
1. Tell your ops guys we know what we're talking about
2. Upgrade mod_jk
3. Set connection_pool_size=1, or, better yet, remove the config
  altogether and let mod_jk determine its own value
4. Remove all timeouts unless you know that you have a misbehaving
  firewall. If you do, enable cping/cpong (the preferred strategy
  by at least one author or mod_jk)

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe
WQCfTNbSLTgNtHcTbTAu5kw5igicNMw=
=0EWv
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message