tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 50306] Detect stuck threads
Date Wed, 29 Dec 2010 10:02:12 GMT
https://issues.apache.org/bugzilla/show_bug.cgi?id=50306

--- Comment #7 from TomLu <thomaslu@tihlde.org> 2010-12-29 05:02:04 EST ---
Created an attachment (id=26451)
 --> (https://issues.apache.org/bugzilla/attachment.cgi?id=26451)
Patch for Tomcat's ThreadPoolExecutor

WebSphere has hung/stuck thread detection logic that has been very useful to
us, both in development and production. But since we are moving away from
WebSphere, it would be nice to have this kind of feature in Tomcat as well.
Some might consider this "stuck thread detection" unnessesary bloat, but we
have found it useful on numerous occasions. 

Applications are never perfect, and often threads will run for too long
(resulting in thread leakage and pool depletion):
-External connections might hang (like URLConnection) [you can configure
read/connect timeout, but not all applications has configured this]
-Loops (can be detected by excessive CPU usage)
-Loops which does not have excessive CPU usage (for example because it uses
external slow IO in each loop iteration, or has Thread.sleep!! (bad coding, but
it can happen))
-Deadlocks.
-Threads waiting on other hung threads.

I have created an experimental patch for Tomcat 7.0.5 that we have been using
in development for a short time now.

The hung thread monitoring does the following:
Monitors active (managed) threads, and issues a warning for each thread that
has been active for more than the configured threshold (seconds).
The message (log.warn) contains the thread name, stack trace, total time
active, and total number of suspected hung threads. 
(In our environment, we use Cron to scan system.out, and send email if a hung
thread is detected.)

A timer thread runs at a configurable interval and checks and reports hung
threads, and threads previously reported to be hung, but have now finished.

It detects hung threads in the HTTP/AJP thread pools. It does not monitor all
JVM threads, but since many user threads are spawned from managed "webrequest"
threads, the HTTP/AJP threads often waits for user threads to finish, so this
patch should give an indication that something is wrong in those cases as well.

If anyone is interested, I've attached the patch. 
This patch is just an example of how such monitoring can be implemented. This
is not production ready code. Performance impacts have not been considered!

The patch is for the following class:

org.apache.tomcat.util.threads.ThreadPoolExecutor

It uses 2 system properties for configuration:

org.apache.catalina.threadmonitor.interval (in seconds. Default 120)
org.apache.catalina.threadmonitor.threshold (in seconds. Default 600)

The monitoring is enabled by default, but can be disabled by setting the
"interval" to a negative value or 0.

As stated, it only monitors managed threads created by Tomcat's HTTP/AJP thread
pools.

Monitoring all JVM threads is also an option, but may require more code and
might lead to many false positives..? The most important thing is to detect
problems with stuck HTTP/AJP threads.

TODOs:

-Should the thread monitor properties be moved to server.xml instead of being
System properties?
-Add JMX notification when hung thread detected/hung thread finished


I have no problem understanding that such a feature might not be seen as useful
for everybody. We will continue to patch our own version of Tomcat anyway :)

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Mime
View raw message