From commons-user-return-18597-apmail-jakarta-commons-user-archive=jakarta.apache.org@jakarta.apache.org Tue Feb 06 23:00:00 2007 Return-Path: Delivered-To: apmail-jakarta-commons-user-archive@www.apache.org Received: (qmail 97953 invoked from network); 6 Feb 2007 22:59:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 6 Feb 2007 22:59:59 -0000 Received: (qmail 67527 invoked by uid 500); 6 Feb 2007 23:00:01 -0000 Delivered-To: apmail-jakarta-commons-user-archive@jakarta.apache.org Received: (qmail 66889 invoked by uid 500); 6 Feb 2007 23:00:00 -0000 Mailing-List: contact commons-user-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Help: List-Post: List-Id: "Jakarta Commons Users List" Reply-To: "Jakarta Commons Users List" Delivered-To: mailing list commons-user@jakarta.apache.org Received: (qmail 66878 invoked by uid 99); 6 Feb 2007 23:00:00 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Feb 2007 15:00:00 -0800 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of mwille@nextusinc.com designates 66.184.96.51 as permitted sender) Received: from [66.184.96.51] (HELO hawaii.nextusinc.com) (66.184.96.51) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Feb 2007 14:59:50 -0800 Received: from NEXTUS-DOM-MTA by hawaii.nextusinc.com with Novell_GroupWise; Tue, 06 Feb 2007 17:59:22 -0500 Message-Id: <45C8C1EE.02EE.000B.0@nextusinc.com> X-Mailer: Novell GroupWise Internet Agent 7.0.1 Date: Tue, 06 Feb 2007 17:59:11 -0500 From: "Michael Wille" To: Subject: [DBCP] Pool Exhausted - But all connections closed - Desperate! Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=__PartC3E7422F.0__=" X-Virus-Checked: Checked by ClamAV on apache.org --=__PartC3E7422F.0__= Content-Type: multipart/alternative; boundary="=__PartC3E7422F.1__=" --=__PartC3E7422F.1__= Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hello All, =20 I'm running into the classic "pool exhausted" problem. Every 36 hours or = so, we hit the wall on our BasicDataSource. But in this case, we *are* = closing our connections. In fact, we've implemented several checks to = make sure all connections are released. =20 I've exhausted all of our avenues so I'm hoping that someone can shed some = light or at least point us in a new direction to try. =20 We have a two app servers, both running Tomcat 5.0.28 (includes dbcp-1.2.1)= . We have one database server, running MySQL 5.0.x. The two tomcat = servers are configured identically and running the same code in a = clustered environment. We are using the Spring Framework in the applicatio= n to load a BasicDataSource. The configuration of it looks like: =20 driverClassName -> com.mysql.jdbc.Driver maxActive -> 100 maxIdle -> 5 maxWait -> 10000 minEvictableIdleTimeMillis -> 5000=20 timeBetweenEvictionRunsMillis -> 5000=20 defaultAutoCommit -> false=20 removeAbandoned -> true removeAbandonedTimeout -> 300 logAbandoned -> true =20 If you'll notice, we have the removeAbandoned feature working. From the = docs, I believe that guarantees that any unclosed connection will be = closed and reported. For my sanity's sake I verified our server logs that = it is using those settings and found: =20 AbandonedObjectPool is used (org.apache.commons.dbcp.AbandonedObjectPool@1d= 709a5) LogAbandoned: true RemoveAbandoned: true RemoveAbandonedTimeout: 300 Now after production has been running for about 24 hours, one of the app = servers has skewed its active connections to up around 78 connections. It = has been holding steady at 70 connections since this morning. (I'm = inspecting these values through JMX) However, there are only 10 active = sessions for the application on that server! Then, when I examine the = database's active connections, it reports only 27 connections open for = that app server. (Also, it has been holding steady at around 20 connection= s all day) That seems like a strange disconnect! =20 And finally, I search through the entire logs for our runaway app server. = I search both the custom log file of our application, and catalina.out of = tomcat. I'm looking for a statement from removeAbandoned about unclosed = connections like: =20 DBCP object created 2007-02-06 17:30:50 by the following code was never = closed: =20 However, this statement (only searching on "DBCP object") never appears in = our log files. So it appears that we are indeed being responsible with = our db connections. It should be noted that we also have custom code that = reports any connections that are still active after a timeout of X = minutes. Essentially, its the same as the configuration to track = abandoned connections. We added this code before we found the nice = configuration options above. =20 I've attached a screen shot that shows the MySQL Admin connections screen = overlaid on JConsole. JConsole is showing both app servers and the graph = of their active connections. Both servers have not been rebooted since = yesterday afternoon. =20 Please Help! We are desperate! We have been trying to troubleshoot this = problem for the past 2 months. =20 Kind Regards, =20 Mike =20 P.S. - though the current screen shot shows only one app server exhibiting = the problem, it is not limited to one or the other and sometimes both = servers hit the limit at close to the same time. Also, the top graph in = the screen shot is not normalized to the one in the bottom. Unfortunately,= JConsole died on me and I had to restart that server's connection. --=__PartC3E7422F.1__= Content-Type: text/html; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Content-Description: HTML
Hello All,
 
I'm running into the classic "pool exhausted" problem.  Every 36 = hours or so, we hit the wall on our BasicDataSource.  But in this = case, we *are* closing our connections.  In fact, we've implemented = several checks to make sure all connections are released.
 
I've exhausted all of our avenues so I'm hoping that someone can shed = some light or at least point us in a new direction to try.
 
We have a two app servers, both running Tomcat 5.0.28 (includes = dbcp-1.2.1).  We have one database server, running MySQL = 5.0.x.  The two tomcat servers are configured identically and running = the same code in a clustered environment.  We are using the Spring = Framework in the application to load a BasicDataSource.  The = configuration of it looks like:
 
driverClassName -> com.mysql.jdbc.Driver
maxActive -> = 100
maxIdle -> 5
maxWait -> 10000
minEvictableIdleTimeMillis= -> 5000 
timeBetweenEvictionRunsMillis -> 5000 
defa= ultAutoCommit -> false 
removeAbandoned -> true
removeAban= donedTimeout -> 300
logAbandoned -> true
 
If you'll notice, we have the removeAbandoned feature working.  = >From the docs, I believe that guarantees that any unclosed connection will = be closed and reported.  For my sanity's sake I verified our server = logs that it is using those settings and found:
 
AbandonedObjectPool is used (org.apache.commons.dbcp.AbandonedObjectPool@1= d709a5)
   LogAbandoned: true
   RemoveAbando= ned: true
   RemoveAbandonedTimeout: 300
Now after production has been running for about 24 hours, one of the = app servers has skewed its active connections to up around 78 connections.&= nbsp; It has been holding steady at 70 connections since this = morning.  (I'm inspecting these values through JMX)  = However, there are only 10 active sessions for the application = on that server!  Then, when I examine the database's active connection= s, it reports only 27 connections open for that app server.  (Also, = it has been holding steady at around 20 connections all day)  That = seems like a strange disconnect!
 
And finally, I search through the entire logs for our runaway app = server.  I search both the custom log file of our application, and = catalina.out of tomcat.  I'm looking for a statement from removeAbando= ned about unclosed connections like:
 
DBCP object created 2007-02-06 17:30:50 by the following code was = never closed:
 
However, this statement (only searching on "DBCP object") never = appears in our log files.  So it appears that we are indeed being = responsible with our db connections.  It should be noted that we also = have custom code that reports any connections that are still active after = a timeout of X minutes.  Essentially, its the same as the = configuration to track abandoned connections.  We added this code = before we found the nice configuration options above.
 
I've attached a screen shot that shows the MySQL Admin connections = screen overlaid on JConsole.  JConsole is showing both app servers = and the graph of their active connections.  Both servers have not = been rebooted since yesterday afternoon.
 
Please Help!  We are desperate!  We have been trying to = troubleshoot this problem for the past 2 months.
 
Kind Regards,
 
Mike
 
P.S. - though the current screen shot shows only one app server = exhibiting the problem, it is not limited to one or the other and = sometimes both servers hit the limit at close to the same time.  = Also, the top graph in the screen shot is not normalized to the one in the = bottom.  Unfortunately, JConsole died on me and I had to restart that = server's connection.
--=__PartC3E7422F.1__=-- --=__PartC3E7422F.0__= Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org For additional commands, e-mail: commons-user-help@jakarta.apache.org --=__PartC3E7422F.0__=--