Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 37496 invoked from network); 8 Apr 2009 19:38:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 8 Apr 2009 19:38:31 -0000 Received: (qmail 89596 invoked by uid 500); 8 Apr 2009 19:38:30 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 89528 invoked by uid 500); 8 Apr 2009 19:38:30 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 89518 invoked by uid 99); 8 Apr 2009 19:38:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 19:38:30 +0000 X-ASF-Spam-Status: No, hits=3.7 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com designates 209.85.220.174 as permitted sender) Received: from [209.85.220.174] (HELO mail-fx0-f174.google.com) (209.85.220.174) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 19:38:20 +0000 Received: by fxm22 with SMTP id 22so351476fxm.38 for ; Wed, 08 Apr 2009 12:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=B12rSwYiC2JjsXtbQTBQcKLgqS6r2SsnlQWjl01gmF0=; b=vMwIxVlheM5Ab2Sa5y2ukzeljRIsO7hHjKBXeX24M0UlQI6wT+pXtceEG1lk6L/MYE fvJ78yzhyaZV4qOgDtloskDu7Iflc55wegqm63TpV6rrzfbAtv86xu5N+QLkyWGkOAbA makyb1S6OxwN9byXG/wdOE765oeHudPHqDt4o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=s5wjmLXoWJ0j0aJvq+VZGVE77es6JVK+MjQNX7Rqd1kpC32qxIrtaU/NitgOhAxodo R2aqTjaNbvXQdI/vygAnVaAu3WHJysB0ETk11bZ50SkYhdkrtYzRIFc2jHeFTFAkJAf/ iHd7nk3cHKnNf2RkNtMSskNDAWOC4VIi6HQxE= MIME-Version: 1.0 Received: by 10.103.6.18 with SMTP id j18mr776697mui.33.1239219477238; Wed, 08 Apr 2009 12:37:57 -0700 (PDT) In-Reply-To: <22939332.post@talk.nabble.com> References: <22440338.post@talk.nabble.com> <7b3355cb0903101217y4db171aek4c8d9d9a98dd735a@mail.gmail.com> <22445136.post@talk.nabble.com> <7b3355cb0903101724w7728fb0cl974b6325dff3365@mail.gmail.com> <22466448.post@talk.nabble.com> <3a73c17c0903120307x1cdd4deex78943aadf013cfcb@mail.gmail.com> <22939332.post@talk.nabble.com> Date: Wed, 8 Apr 2009 20:37:57 +0100 Message-ID: <3a73c17c0904081237g40108485v76efd42df7a7ebfa@mail.gmail.com> Subject: Re: Is there JDBC master/slave retry logic if database goes down? From: Gary Tully To: users@activemq.apache.org Content-Type: multipart/alternative; boundary=001636416899bd544804671044cf X-Virus-Checked: Checked by ClamAV on apache.org --001636416899bd544804671044cf Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to set a locker instance. see: http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker) Otherwise, yea, as you describe, derive and inject using spring via the XML configuration. hope this helps, Gary. 2009/4/7 magellings > > How do you exactly provide your own implementation? > > On the JDBCPersistenceAdaptor the createDatabaseLocker will always use the > DefaultDatabaseLocker: > > protected DatabaseLocker createDatabaseLocker() throws IOException { > return new DefaultDatabaseLocker(getLockDataSource(), getStatements()); > } > > Can I provide my own implemenation declaratively through configuration > provided I write a class deriving from JDBCPersistenceAdaptor of which > overrides the createDatabaseLocker class? > > > Gary Tully wrote: > > > > One thing to note: you can provide your own implementation of the > > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement > > keepAlive to be aware of your database backup policy. > > > > If you go down this route and conceive a better default or optional > > strategy, feel free to submit it as a patch. > > > > > > > > 2009/3/11 magellings > > > >> > >> Well it was butt-simple to get working like the c3p0 documentation said. > >> I > >> upped the delay to 20s, unfortunately the master broker still failed. > >> > >> I'm kind of wondering how the slave would respond. My guess is it would > >> probably time out too. > >> > >> Any thoughts? > >> > >> See further below for the errors. > >> > >> > >> > C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat > >> xbean:activemq.xml > >> ACTIVEMQ_HOME: > >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\.. > >> ACTIVEMQ_BASE: > >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\.. > >> Loading message broker from: xbean:activemq.xml > >> INFO DefaultCamelContext - JMX enabled. Using > >> InstrumentationLifecycleStrategy. > >> INFO MLog - MLog clients using log4j logging. > >> INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built > >> 21-May-2007 15:04:56; debug? true; trace: 10] > >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool... > >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, > >> acquireRetryAttemp > >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false, > >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false, > >> checkoutTimeout - > >> > 30000, connectionCustomizerClassName -> null, > connectionTesterClassName > >> -> > >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> > >> 2rvy1 > >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false, > >> description -> null, driverClass -> > >> com.microsoft.sqlserver.jdbc.SQLServerDri > >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8, > >> idleConnectionTestPerio > >> d -> 30, initialPoolSize -> 10, jdbcUrl -> > >> jdbc:sqlserver://QG076619.qg.com:1433 > >> ;databaseName=activedb;user=sa;password=pa$$word, > >> maxAdministrativeTas > >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30, > >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements -> > >> 200, > >> maxStatementsPerCon > >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3, > >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, > >> properties -> {user=** > >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> > >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> > >> 0, > >> use > >> sTraditionalReflectiveProxies -> false ] > >> INFO BrokerService - Using Persistence Adapter: > >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [ > >> acquireIncrement > >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000, > >> autoCommitOnClose -> false, automaticTestTable -> con_test, > >> breakAfterAcquireFailure -> > >> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null, > >> connectionTesterClassName -> > >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, > >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8, > >> debugUnreturnedConnectionStackTraces -> false, description -> null, > >> driverClass -> com.microsoft.sql > >> server.jdbc.SQLServerDriver, factoryClassLocation -> null, > >> forceIgnoreUnresolvedTransactions -> false, identityToken -> > >> 2rvy117z1wqpy2hxzkebp|18105e8, > >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl -> > >> jdbc:sqlserver://QG076619.qg.com:1433 > >> ;databaseName=activedb;user=sa;password=pa$$wo > >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime > -> > >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements > >> -> > >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10, > numHelperThreads > >> -> > >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> > null > >> , properties -> {user=******, password=******}, propertyCycle -> 0, > >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, > >> unreturnedCon > >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]) > >> INFO JDBCPersistenceAdapter - Database driver recognized: > >> [microsoft_sql_server_2005_jdbc_driver] > >> INFO DefaultDatabaseLocker - Attempting to acquire the > >> exclusive > >> lock to become the Master broker > >> INFO DefaultDatabaseLocker - Becoming the master on > dataSource: > >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, > >> acquireRe > >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> > >> false, > >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false, > >> checkout > >> Timeout -> 30000, connectionCustomizerClassName -> null, > >> connectionTesterClassName -> > >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName > >> -> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces > >> -> > >> false, description -> null, driverClass -> > >> com.microsoft.sqlserver.jdbc.SQL > >> ServerDriver, factoryClassLocation -> null, > >> forceIgnoreUnresolvedTransactions -> false, identityToken -> > >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection > >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl -> > >> jdbc:sqlserver://QG076619.qg.com:1433 > >> ;databaseName=activedb;user=sa;password=pa$$word, > >> maxAdminist > >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30, > >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements -> > >> 200, > >> maxStateme > >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3, > >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, > >> properties -> > >> {user=******, password=******}, propertyCycle -> 0, > >> testConnectionOnCheckin > >> -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout > >> -> 0, usesTraditionalReflectiveProxies -> false ] > >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker > >> (master) is starting > >> INFO BrokerService - For help or more information > >> please > >> see: http://activemq.apache.org/ > >> INFO TransportServerThreadSupport - Listening for connections at: > >> tcp://QG076619:61616 > >> INFO TransportConnector - Connector openwire Started > >> INFO TransportServerThreadSupport - Listening for connections at: > >> ssl://QG076619:61617 > >> INFO TransportConnector - Connector ssl Started > >> INFO TransportServerThreadSupport - Listening for connections at: > >> stomp://QG076619:61613 > >> INFO TransportConnector - Connector stomp Started > >> INFO TransportServerThreadSupport - Listening for connections at: > >> xmpp://QG076619:61222 > >> INFO TransportConnector - Connector xmpp Started > >> INFO BrokerService - ActiveMQ JMS Message Broker > >> (master, > >> ID:QG076619-3632-1236812198396-0:0) started > >> INFO log - Logging to > >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via > >> org.mortbay.log.Slf4jLog > >> INFO log - jetty-6.1.9 > >> INFO WebConsoleStarter - ActiveMQ WebConsole initialized. > >> INFO /admin - Initializing Spring > >> FrameworkServlet > >> 'dispatcher' > >> INFO log - ActiveMQ Console at > >> http://0.0.0.0:8162/admin > >> INFO log - ActiveMQ Web Demos at > >> http://0.0.0.0:8162/demo > >> INFO log - RESTful file access application > at > >> http://0.0.0.0:8162/fileserver > >> INFO log - Started > >> SelectChannelConnector@0.0.0.0:8162 > >> WARN BrokerRegistry - Broker localhost not started so > >> using > >> master instead > >> INFO TransportConnector - Connector vm://localhost Started > >> > >> > >> Then BOOM! Although behavior is different. Now it takes longer to > stop. > >> :) "The connection to '/10.0.1.5:3333' is taking a long time to > >> shutdown." > >> If you look closely you can see activeMQ is timing out. > >> > >> > >> > >> > >> ERROR DefaultDatabaseLocker - Failed to update database lock: > >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe > >> et write error > >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by > >> peer: > >> socket write error > >> at > >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown > >> Source) > >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source) > >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source) > >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown > >> Source) > >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown > >> Source) > >> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown > >> Source) > >> at > >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown > >> Source) > >> at > >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown > >> Source) > >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown > Source) > >> at > >> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown > >> Source) > >> at > >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown > >> Source) > >> at > >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown > >> Source) > >> at > >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown > >> Source) > >> at > >> > >> > com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) > >> at > >> > >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118) > >> at > >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481) > >> at > >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >> at > >> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > >> at > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > >> at > >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > >> at > >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) > >> at > >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) > >> at > >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> at > >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:619) > >> INFO GooGooStatementCache - Problem with checked-in > Statement, > >> discarding. > >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is > >> closed. > >> at > >> > com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown > >> Source) > >> at > >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown > >> Source) > >> at > >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown > >> Source) > >> at > >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown > >> Source) > >> at > >> > >> > com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604) > >> at > >> > >> > com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236) > >> at > >> > >> > com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282) > >> at > >> > >> > com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807) > >> at > >> > >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127) > >> at > >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481) > >> at > >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >> at > >> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > >> at > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > >> at > >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > >> at > >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) > >> at > >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) > >> at > >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> at > >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:619) > >> INFO JDBCPersistenceAdapter - No longer able to keep the > >> exclusive > >> lock so giving up being a master > >> INFO BrokerService - ActiveMQ Message Broker (master, > >> ID:QG076619-3332-1236811661934-0:0) is shutting down > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> WARN ThreadPoolAsynchronousRunner - > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766 > >> -- APPARENT DEADLOCK!!! Crea > >> rgency threads for unassigned pending tasks! > >> WARN ThreadPoolAsynchronousRunner - > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766 > >> -- APPARENT DEADLOCK!!! Comp > >> tus: > >> Managed Threads: 3 > >> Active Threads: 3 > >> Active Tasks: > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a > >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097 > >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49 > >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea > >> Pending Tasks: > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087 > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561 > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141 > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6 > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945 > >> > >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3 > >> > >> > com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc > >> Pool thread stack traces: > >> > >> > >> > Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main] > >> java.lang.Thread.sleep(Native Method) > >> > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805) > >> > >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) > >> > >> > >> > Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main] > >> java.lang.Thread.sleep(Native Method) > >> > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805) > >> > >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) > >> > >> > >> > Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main] > >> java.lang.Thread.sleep(Native Method) > >> > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805) > >> > >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) > >> > >> > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> INFO TransportConnection - The connection to '/ > 10.0.1.5:3333' > >> is > >> taking a long time to shutdown. > >> ERROR QueueStorePrefetch - Failed to get message count > >> java.io.IOException: An attempt by a client to checkout a Connection has > >> timed out. > >> at > >> > >> > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45) > >> at > >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61) > >> at > >> > >> > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687) > >> at > >> > >> > org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208) > >> at > >> > >> > org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79) > >> at > >> > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63) > >> at > >> > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212) > >> at > >> > >> > org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257) > >> at > >> > org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308) > >> at > >> > >> > org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324) > >> at > >> > >> > org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392) > >> at > >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114) > >> at > >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114) > >> at > >> > >> > org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222) > >> at > >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114) > >> at > >> > >> > org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000) > >> at > >> > >> > org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925) > >> Caused by: java.sql.SQLException: An attempt by a client to checkout a > >> Connection has timed out. > >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106) > >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65) > >> at > >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527) > >> at > >> > >> > com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128) > >> at > >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54) > >> ... 20 more > >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed > >> out > >> while waiting to acquire a resource from com.mchange.v2.resourcepo > >> ResourcePool@d964af -- timeout at awaitAvailable() > >> at > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317) > >> at > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557) > >> at > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477) > >> at > >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525) > >> ... 22 more > >> WARN TransportConnection - Failed to remove consumer: > >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason: > >> java.lang.RuntimeException > >> o.IOException: An attempt by a client to checkout a Connection has timed > >> out. > >> java.lang.RuntimeException: java.io.IOException: An attempt by a client > >> to > >> checkout a Connection has timed out. > >> at > >> > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66) > >> at > >> > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212) > >> at > >> > >> > org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257) > >> at > >> > org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308) > >> at > >> > >> > org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324) > >> at > >> > >> > org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392) > >> at > >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114) > >> at > >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114) > >> at > >> > >> > org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222) > >> at > >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114) > >> at > >> > >> > org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000) > >> at > >> > >> > org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74) > >> at > >> > >> > org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925) > >> Caused by: java.io.IOException: An attempt by a client to checkout a > >> Connection has timed out. > >> at > >> > >> > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45) > >> at > >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61) > >> at > >> > >> > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687) > >> at > >> > >> > org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208) > >> at > >> > >> > org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79) > >> at > >> > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63) > >> ... 16 more > >> Caused by: java.sql.SQLException: An attempt by a client to checkout a > >> Connection has timed out. > >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106) > >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65) > >> at > >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527) > >> at > >> > >> > com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128) > >> at > >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54) > >> ... 20 more > >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed > >> out > >> while waiting to acquire a resource from com.mchange.v2.resourcepo > >> ResourcePool@d964af -- timeout at awaitAvailable() > >> at > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317) > >> at > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557) > >> at > >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477) > >> at > >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525) > >> ... 22 more > >> INFO TransportConnector - Connector openwire Stopped > >> > >> > >> bsnyder wrote: > >> > > >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings > >> wrote: > >> >> > >> >> Thanks for the response Bruce. :) > >> >> > >> >> To my knowledge we aren't using a connection pooler. We're just > >> >> configuring > >> >> the activemq xml and letting activeMQ do its thing. One of my > >> colleagues > >> >> mentioned SNAC since we are using Sql Server, not sure if that is > >> similar > >> >> to > >> >> a connection pooler or not. > >> > > >> > The job of a JDBC connection pooler is to keep a pool of JDBC > >> > connections available for use by a Java application. IIRC, SNAC stands > >> > for SQL Native Client which has nothing to do with a JDBC connection > >> > pooler. > >> > > >> >> In our scenario during a Sql Server failover it may take 15 minutes. > >> We > >> >> aren't using a high availability cluster. Does a connection pooler > >> give > >> >> the > >> >> capability of dealing with this so that ActiveMQ doesn't shutdown? > >> > > >> > One of the options in a good JDBC connection pooler is the ability to > >> > test JDBC connections as it hands them out to the Java application. If > >> > the connection is stale for any reason the JDBC connection pooler will > >> > evict that connection from the pool and try the next one until it > >> > reaches one that is alive. To my knowledge, some JDBC connection > >> > poolers provide a the ability to configure some attributes like number > >> > of retries, etc. E.g., here is some information about what c3p0 > >> > offers: > >> > > >> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery > >> > > >> > The one big caveat I will mention is that longer outage periods such > >> > as the 15 minute outage you mention are not typically handled by a > >> > JDBC connection pooler as they're mainly geared toward handling > >> > momentary outages. As an example, notice that the default value for > >> > acquireRetryDelay in c3p0 is only one second: > >> > > >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay > >> > > >> > But you could certainly configure this to any value you want. I > >> > encourage you to test this in your environment before deploying to > >> > production. > >> > > >> > Bruce > >> > -- > >> > perl -e 'print > >> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E >> > );' > >> > > >> > Apache ActiveMQ - http://activemq.apache.org/ > >> > Apache Camel - http://camel.apache.org/ > >> > Apache ServiceMix - http://servicemix.apache.org/ > >> > > >> > Blog: http://bruceblog.org/ > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > > http://blog.garytully.com > > > > Open Source SOA > > http://FUSESource.com > > > > > > -- > View this message in context: > http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > -- http://blog.garytully.com Open Source SOA http://FUSESource.com --001636416899bd544804671044cf--