ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Torsten.Ro...@teliasonera.com>
Subject DBCP Deadlock? [Was: iBatis settings and datasource configuration]
Date Tue, 27 Nov 2007 22:35:47 GMT
A follow up on this issue:

We replaced the SimpleDataSource with Apache Commons DBCP but see the same issue: The pool
is getting in some kind of deadlock state where threads are blocked while trying to get a
lock on the pool in order to return or borrow a connection to or from the pool. Please see
the thread dump below.

I'd assume that both borrowObject() and returnObject() always complete promptly and therefore
such a deadlock never occurs, and that in case the database can't keep up the pool rather
gets exhausted, but the behaviour seen here I really don't understand.

It happens usually after a couple of days at peak times where also the database is really
loaded.

These are our current iBatis settings:

    <settings
        cacheModelsEnabled="true"
        enhancementEnabled="true"
        maxRequests="320"
        maxSessions="128"
        maxTransactions="64"/>

And the DBCP settings:

	maxActive=64
	maxIdle=16
	maxWait=1000
	validationQuery=select * from dual

"TP-Processor483" daemon prio=10 tid=0x01fe5000 nid=0x11f7 waiting for monitor entry [0x8d87e000..0x8d881788]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:916)
	- waiting to lock <0x9ddf7460> (a org.apache.commons.pool.impl.GenericObjectPool)
	at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:87)
	- locked <0xf83f9eb8> (a org.apache.commons.dbcp.PoolableConnection)
	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
	at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.close(JdbcTransaction.java:81)
	at com.ibatis.sqlmap.engine.transaction.TransactionManager.end(TransactionManager.java:110)

"TP-Processor382" daemon prio=10 tid=0x026f6000 nid=0x25a waiting for monitor entry [0x7de7f000..0x7de81808]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
	- waiting to lock <0x9ddf7460> (a org.apache.commons.pool.impl.GenericObjectPool)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.init(JdbcTransaction.java:48)
	at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:89)

Torsten

-----Original Message-----
From: Torsten.Romer@teliasonera.com [mailto:Torsten.Romer@teliasonera.com] 
Sent: 25. september 2007 22:50
To: user-java@ibatis.apache.org
Subject: RE: iBatis settings and datasource configuration


The app is quite critical, it's our customer activation system. It is pretty serious, and
the load is considerable.

We are already considering to go for Apache commons DBCP instead, since we have only good
experience with it in a number of other apps. Your reply supports this - thanks for that!
:-)

Since we have a new release very soon, we'll go with the adjusted settings for now, but I
guess from next release we'll use DBCP. After all, the name SimpleDataSource kind of suggests
that it may not be suitable for larger apps...

Thanks for your helpful advice!

Torsten
 

From: Adam Zimowski [mailto:zimowski74@gmail.com] 
Sent: 25. september 2007 21:27

How critical is your app? What's the peak load? If this is a serious enterprise app, I'd consider
using alternate data source:

* From Developer Guide (I'm sure you've seen that) *
Note!: SimpleDataSource is quite convenient, efficient and effective.
However, for large enterprise or mission critical applications, it is recommended that you
use an enterprise level DataSource implementation (such as those that come with App Servers
and commercial O/R mapping tools).

Reducing Pool.MaximumCheckoutTime may help to narrow down the issue, as checked out connections
will be eligible for pool sooner than you may have specified currently. Also, advising your
DAOs with something like Spring AOP or AspectJ to log meaningful usage statistics may help
to provide some verbosity to how your DAO layer is utilized by your web app, especially in
regards to the load it may be experiencing at different times.

In my opinion, the most straightforward way to attack the problem is to try a different data
source implementation. For many years I've been very successfull with Apache commons DBCP,
and given the warning about SimpleDataSource usage in iBatis documentation I'd definitely
try alternate implementation before investing too much time and effort on what may be a dead-end.

In regards to settings, your suggestion to try higher maxRequests is what I would also try.
I'm not sure if reducing max transactions will make much difference, because your db server
isn't likely to experiencing half of that. You have max transactions at 64 but it probably
never even reaches that limit.

One last thing - although it may be an overkill - is to enable iBatis debugging via in your
logger, which may provide some interesting hints. That however, may come at cost of significant
slowdown to your app (depending on the load which I don't know what it is).

Regards,
-adam

On 9/24/07, Torsten.Romer@teliasonera.com <Torsten.Romer@teliasonera.com> wrote:
> Our J2SE webapp using iBatis with SimpleDataSource running in Apache Tomcat occasionally
starts to hang. It hits the roof of max. Nr. of threads and looking at the thread dump, there
are many threads that seem blocked trying to get and return(!) connections from and to the
pool:
>
> "TP-Processor406" daemon prio=10 tid=0x005fbc00 nid=0xda4 waiting for monitor entry [0x7ce7f000..0x7ce81788]
>    java.lang.Thread.State: BLOCKED (on object monitor)  at 
> com.ibatis.common.jdbc.SimpleDataSource.pushConnection(SimpleDataSource.java:521)
>   - waiting to lock <0x9e9d37f0> (a java.lang.Object)  at 
> com.ibatis.common.jdbc.SimpleDataSource.access$100(SimpleDataSource.ja
> va:52)  at 
> com.ibatis.common.jdbc.SimpleDataSource$SimplePooledConnection.invoke(
> SimpleDataSource.java:954)
>  at $Proxy21.close(Unknown Source)
> [...]
>
> It almost looks to me as if SimpleDataSource blocks threads from returning connections
to the pool, while other threads are trying to get connections from it - some kind of deadlock
scenario?
>
> Assuming that my observation is wrong, I wonder if our configuration is good. We currently
have:
>
> iBatis settings
>
>         maxRequests="128"
>         maxSessions="64"
>         maxTransactions="64"
>
> and datasource settings
>
>         maxActive=50
>         maxIdle=15
>
> Isn't maxTransactions too high? Besides the rules pointed out in the documentation, are
there rules on how these settings should relate to the datasource settings? Something like
maxActive should be at least maxSessions or something like that?
>
> Would these settings be any good?
>
>         maxRequests="320"
>         maxSessions="64"
>         maxTransactions="32"
>
> Torsten
>

Mime
View raw message