activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raphaël Ramirez (JIRA) <j...@apache.org>
Subject [jira] Created: (AMQ-2296) Connection pool exhausted in JDBC master-slave when slave tries to acquire lock
Date Thu, 18 Jun 2009 10:29:35 GMT
Connection pool exhausted in JDBC master-slave when slave tries to acquire lock
-------------------------------------------------------------------------------

                 Key: AMQ-2296
                 URL: https://issues.apache.org/activemq/browse/AMQ-2296
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.2.0
         Environment: Windows XP or Linux (Debian 4.1) / Java 1.6  / MySQL 5.0.67 / Spring
2.5.6 / Apache Commons DBCP 1.2.2 + Commons Pool 1.3 / MySQL connector Java 5.1.7
            Reporter: Raphaël Ramirez


When the slave broker fails to acquire the database lock, which is the normal behavior, we
have this (this exception is normal and may be caught by a ExceptionHandler to avoid filling
the logs):

09/06/2009 17:41:05 INFO  org.apache.activemq.broker.BrokerService  [http-8081-1] | Using
Persistence Adapter: JDBCPersistenceAdaptor(app.db.ConfigurableDbcpDataSource@77baf4)
09/06/2009 17:41:06 INFO  org.apache.activemq.store.jdbc.JDBCPersistenceAdapter  [http-8081-1]
| Database driver recognized: [mysql-ab_jdbc_driver]
12/06/2009 18:07:18 INFO  org.apache.activemq.store.jdbc.DefaultDatabaseLocker  [http-8081-1]
| Attempting to acquire the exclusive lock to become the Master broker
12/06/2009 18:09:19 ERROR org.apache.activemq.store.jdbc.DefaultDatabaseLocker  [http-8081-1]
| Failed to acquire lock: java.sql.SQLException: Lock wait timeout exceeded; try restarting
transaction
java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
...

But after 200 (=size of my DBCP connection pool) attempts, I get the following stacktrace
:

13/06/2009 03:56:42 ERROR org.apache.activemq.store.jdbc.DefaultDatabaseLocker  [http-8081-1]
| Failed to acquire lock: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection,
pool error Timeout waiting for idle object
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting
for idle object
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at org.apache.activemq.store.jdbc.DefaultDatabaseLocker.start(DefaultDatabaseLocker.java:63)
	at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:179)
	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:458)
	at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:52)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1369)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
	at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3934)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4429)
	at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1249)
	at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:612)
	at org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:136)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:825)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	... 42 more

Here are my JDBC connection parameters :
persistenceDatabase.url -> jdbc:mysql://xxxxxx/msgbroker?relaxAutoCommit=true
persistenceDatabase.connectionPool.maxActive -> 200
persistenceDatabase.connectionPool.maxWait -> 1000
persistenceDatabase.connectionPool.maxIdle -> 10
persistenceDatabase.connectionPool.validationQuery -> select 1
persistenceDatabase.connectionPool.testOnBorrow -> false
persistenceDatabase.connectionPool.testOnReturn -> false

What happens is that a new connection is borrowed from the pool each time the slave is trying
to acquire the exclusive lock, instead of using the existing connection.
This leads to a connection pool leak. See org.apache.activemq.store.jdbc.DefaultDatabaseLocker
(line 63) :

connection = dataSource.getConnection();

Changing it to the following code solved the issue :

            	if (connection==null || connection.isClosed()) {
            		connection = dataSource.getConnection();
                	connection.setAutoCommit(false);
            	}

Now, an existing connection is used if it is available.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message