commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Torsten.Ro...@teliasonera.com>
Subject RE: DBCP Locking Up?
Date Wed, 05 Dec 2007 00:37:35 GMT
Hello Bill,

A connection leak, interesting. Maybe we have a similar issue.

It's a bit tricky since we are using iBatis which is doing a lot of things behind the scenes,
but I think we could also implement such a logging wrapper around the pool.

However it seems that our pool exhaustion is caused by a combination of poor database performance
and high concurrency; the webapp is using Ajax and there can be many (concurrent?) database
requests caused by a single user action.

Thanks for sharing your experience, I'll get back with our findings!

Torsten

-----Original Message-----
From: Wm.A.Stafford [mailto:stafford@marine.rutgers.edu] 
Sent: 4. december 2007 19:56
To: Jakarta Commons Users List
Subject: Re: DBCP Locking Up?


Torsten,

    We were seeing the same behavior in a lightly loaded application 
when the connection pool was exhausted. I spent several days trying to 
figure out what was happening.  A google search turned up several forum 
and mailing list postings reporting the same problem.  I eventually came 
to the conclusion that the deadlock is some sort of undocumented 
behavior of DBCP that occurs when the pool can no longer fulfill a 
request for a connection.  This was with dbcp-1.2.2 and commons-pool-1.3.  

    I was pretty sure our pool exhaustion was caused by a connection 
leak because we used dbcp to replace a non-pooling DataSource.  So 
connections which would have formerly been garbage collected if not 
closed now were leaked.   We located the leaks by wrapping DBCP with 
class that logged  the state of the pool for every open and close, ex: 
log.info(" active connections=" + pool.getNumActive() + " idle 
connections=" + pool.getNumIdle()) ;

    Eliminating the connection leak eliminated our deadlocks but I never 
figured out what the underlying DBCP problem is. 

    -=bill

Torsten.Romer@teliasonera.com wrote:
> We are using DBCP with iBatis in a JSF/Ajax webapp running in Tomcat with an Oracle database.
>
> The application has quite high load and concrrency, and the database is very loaded.
The application is running fine for some days but then freezes at peak hours.
>
> When I look at a thread dump made while the app hangs, I can see as many threads as we
have connections in the pool (64) in BLOCKED state while calling either borrowObject() or
returnObject(), please see the two examples below. It looks like some kind of deadlock and
the only way out of it seems to be a restart of Tomcat.
>
> First I thought this is normal with the WHEN_EXHAUSTED_LOCK action, but since we have
maxWait set to 1000 I'd not expect this kind of blocking to happen.
>
> I tried to reproduce the behavior with a small test app, but here the threads go in TIMED_WAIT
state when the pool is exhausted and become RUNNABLE again as soon as some connection became
available again.
>
> I really don't understand what is happening and why the pool gets locked up that way.
>
> Any hint/suggestion welcome!
>
> Torsten
>
> "TP-Processor483" daemon prio=10 tid=0x01fe5000 nid=0x11f7 waiting for monitor entry
[0x8d87e000..0x8d881788]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> 	at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:916)
> 	- waiting to lock <0x9ddf7460> (a org.apache.commons.pool.impl.GenericObjectPool)
> 	at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:87)
> 	- locked <0xf83f9eb8> (a org.apache.commons.dbcp.PoolableConnection)
> 	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
> 	at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.close(JdbcTransaction.java:81)
> 	at com.ibatis.sqlmap.engine.transaction.TransactionManager.end(TransactionManager.java:110)
>
> "TP-Processor382" daemon prio=10 tid=0x026f6000 nid=0x25a waiting for monitor entry [0x7de7f000..0x7de81808]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> 	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
> 	- waiting to lock <0x9ddf7460> (a org.apache.commons.pool.impl.GenericObjectPool)
> 	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
> 	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
> 	at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.init(JdbcTransaction.java:48)
> 	at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:89)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>   

-- 

Wm.A.Stafford

Principal Engineer

The Institute of Marine and Coastal Sciences
Rutgers University
732-932-6555 x367
stafford@marine.rutgers.edu

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


Mime
View raw message