tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kapilok <kapi...@yahoo.com>
Subject Re: Recover from SEVERE: All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status
Date Fri, 09 Apr 2010 13:46:00 GMT



Christopher Schultz-2 wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Kapilok,
> 
> On 4/6/2010 11:19 AM, kapilok wrote:
>> Does Tomcat guarantee recovery in case of this Error?
> 
> The JVM/Tomcat should "recover" in the sense that service threads that
> complete their work should go back into the thread pool and accept new
> requests.
> 
> Unfortunately, your webapp can significantly impact the ability for
> those threads to complete their request handling.
> 
>> Can you point me to the documentation?
> 
> There is no documentation that covers this "recovery": it's just a
> feature of using thread pools.
> 
>> Please suggest a way to test / validate the recovery.
> 
> First, figure out what the real problem is. The symptom is obvious (see
> your subject line).
> 
> To see what those threads are doing, allow this situation to occur, then
> do the following:
> 
> 1. Perform whatever magic you need to "solve" the "database overload"
> issue
> 
> 2. Wait a minute or two to give your webapp a chance to breathe
> 
> 3. Perform a thread dump on your webapp
> (http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F)
> 
> 4. Wait 30 seconds
> 
> 5. Perform another thread dump on your webapp
> 
> 6. Inspect the thread dump(s) to see what your threads are stuck doing.
> If they're stuck in your webapp's code, go fix that and try again. If
> they're stuck in Tomcat's code, post back and we'll take a look.
> 
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAku7WkgACgkQ9CaO5/Lv0PA/rgCffQZFnykd/0eXKdbOxlY/6XA9
> TskAoLNKqjjcJbrAyoKeqT5J6TRnsVKX
> =KvA6
> -----END PGP SIGNATURE-----
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> 

Chris,

I think the thread dump does suggest that it is the webapp code (c3p0
connection pool) and not tomcat. But Please take a look anyways: (pasted
partial dump only)

Here's my test scenario:
1. Start Tomcat (with maxThreads="40" )
2. Run JMeter Load (40 concurrent with some ramp up)
- All requests succeed
3. Now load the database with some heavy process, so CPU consumption is high
4. Run same jMeter load; now response times are slow
5. Get ThreadPool Full Error; Take Thread Dump
6. Kill all JMeter requests; so Tomcat can breathe
7. Wait 10 minutes. Try login to webapp - Cant login. Browser does not
display Login page or any other page.
8. Take another Thread Dump. Its the same. NOTE: Main Thread and
TP-Processor4 are still running (observed with VisualVM1.1 on JDK 6)
9. Try running jMeter load. No reponse - No requests going through.
10. Waited 30 minutes and had no option but to bounce Tomcat.

Bottom Line: What can I do so I don't have to bounce Tomcat when I run into
this situation?

Partial Thread Dump
*******************
"http-9080-Processor40" daemon prio=6 tid=0x4d1ba000 nid=0x1e0 in
Object.wait() [0x5162e000..0x5162fd94]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x07b12648> (a com.mchange.v2.resourcepool.BasicResourcePool)
	at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
	at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
	- locked <0x07b12648> (a com.mchange.v2.resourcepool.BasicResourcePool)
	at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
	at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
	at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
	at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
	at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
	at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:523)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:587)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:612)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:644)
...

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon
prio=6 tid=0x4dcda400 nid=0x6ac in Object.wait() [0x5068f000..0x5068fb94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0a779ea8> (a
com.mchange.v2.async.ThreadPoolAsynchronousRunner)
	at
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
	- locked <0x0a779ea8> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

   Locked ownable synchronizers:
	- None

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon
prio=6 tid=0x4d261800 nid=0x8c in Object.wait() [0x5063f000..0x5063fa94]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0a779ea8> (a
com.mchange.v2.async.ThreadPoolAsynchronousRunner)
	at
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
	- locked <0x0a779ea8> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

   Locked ownable synchronizers:
	- None

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon
prio=6 tid=0x4cb04800 nid=0x1150 in Object.wait() [0x505ef000..0x505efb14]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0a779ea8> (a
com.mchange.v2.async.ThreadPoolAsynchronousRunner)
	at
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
	- locked <0x0a779ea8> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

   Locked ownable synchronizers:
	- None





-- 
View this message in context: http://old.nabble.com/Recover-from-SEVERE%3A-All-threads-%28400%29-are-currently-busy%2C-waiting.-Increase-maxThreads-%28400%29-or-check-the-servlet-status-tp28150773p28190846.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message