From users-return-18696-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Thu Apr 09 15:47:45 2009 Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 72302 invoked from network); 9 Apr 2009 15:47:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 9 Apr 2009 15:47:45 -0000 Received: (qmail 16540 invoked by uid 500); 9 Apr 2009 15:47:45 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 16499 invoked by uid 500); 9 Apr 2009 15:47:44 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 16489 invoked by uid 99); 9 Apr 2009 15:47:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Apr 2009 15:47:44 +0000 X-ASF-Spam-Status: No, hits=3.7 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com designates 209.85.220.174 as permitted sender) Received: from [209.85.220.174] (HELO mail-fx0-f174.google.com) (209.85.220.174) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Apr 2009 15:47:34 +0000 Received: by fxm22 with SMTP id 22so834013fxm.38 for ; Thu, 09 Apr 2009 08:47:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=XsGX44iJdn3ZNPi5u/5niJy3JBvjqSZYGqice4YyrF0=; b=kZVRyaEN/3sZYmoPUukD/NF2zag0ux4rso+l0HpkZy4CBnCRr+eOMgQq188kd/Sjk5 BiChLcJf5P48jq2YMTdur7b7kGKhcoqaaf01NNA+fPDsXgbC0Y4xTXzUSi+P3AsWPjkk EvuGEk0TKT3dMiDx9KJ4yaEYJmF6y0bvqJUlg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=d/+Y74wJDyhvOP6QpTtajHTsXDtQPDskpIJslVI2W8LY2LIGrO2hV+8So3GGpxzO6J 6AqAroLLtB7aQgQh/PjPzCGqTmLU1/WUTQMK1fJ8ROnTi1CksJwcZl7Av3txj6FCfni9 5WacTqlgZNVzGpBS5rZWLhWg3NtxBYGzvvh20= MIME-Version: 1.0 Received: by 10.103.91.2 with SMTP id t2mr1313445mul.47.1239292032470; Thu, 09 Apr 2009 08:47:12 -0700 (PDT) In-Reply-To: <22973315.post@talk.nabble.com> References: <22440338.post@talk.nabble.com> <7b3355cb0903101217y4db171aek4c8d9d9a98dd735a@mail.gmail.com> <22445136.post@talk.nabble.com> <7b3355cb0903101724w7728fb0cl974b6325dff3365@mail.gmail.com> <22466448.post@talk.nabble.com> <3a73c17c0903120307x1cdd4deex78943aadf013cfcb@mail.gmail.com> <22939332.post@talk.nabble.com> <3a73c17c0904081237g40108485v76efd42df7a7ebfa@mail.gmail.com> <22973315.post@talk.nabble.com> Date: Thu, 9 Apr 2009 16:47:12 +0100 Message-ID: <3a73c17c0904090847o2da119e9s7eb895cd64540714@mail.gmail.com> Subject: Re: Is there JDBC master/slave retry logic if database goes down? From: Gary Tully To: users@activemq.apache.org Content-Type: multipart/alternative; boundary=00163662e5695e4ea40467212950 X-Virus-Checked: Checked by ClamAV on apache.org --00163662e5695e4ea40467212950 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You have got to implement the DataBaseLocker interface in code though or extend the current default to make it behave as you need. 2009/4/9 magellings > > This implies that I'd need to modify activemq source though doesn't it? > > Do you have an example of how it could be done through configuration? > > > Gary Tully wrote: > > > > in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you = to > > set a locker instance. > > see: > > > http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/a= ctivemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache= .activemq.store.jdbc.DatabaseLocker) > > > > Otherwise, yea, as you describe, derive and inject using spring via the > > XML > > configuration. > > > > hope this helps, > > Gary. > > 2009/4/7 magellings > > > >> > >> How do you exactly provide your own implementation? > >> > >> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use > >> the > >> DefaultDatabaseLocker: > >> > >> protected DatabaseLocker createDatabaseLocker() throws IOException { > >> return new DefaultDatabaseLocker(getLockDataSource(), getStatements()= ); > >> } > >> > >> Can I provide my own implemenation declaratively through configuration > >> provided I write a class deriving from JDBCPersistenceAdaptor of which > >> overrides the createDatabaseLocker class? > >> > >> > >> Gary Tully wrote: > >> > > >> > One thing to note: you can provide your own implementation of the > >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement > >> > keepAlive to be aware of your database backup policy. > >> > > >> > If you go down this route and conceive a better default or optional > >> > strategy, feel free to submit it as a patch. > >> > > >> > > >> > > >> > 2009/3/11 magellings > >> > > >> >> > >> >> Well it was butt-simple to get working like the c3p0 documentation > >> said. > >> >> I > >> >> upped the delay to 20s, unfortunately the master broker still faile= d. > >> >> > >> >> I'm kind of wondering how the slave would respond. My guess is it > >> would > >> >> probably time out too. > >> >> > >> >> Any thoughts? > >> >> > >> >> See further below for the errors. > >> >> > >> >> > >> >> > >> > C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activ= emq.bat > >> >> xbean:activemq.xml > >> >> ACTIVEMQ_HOME: > >> >> > C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\.. > >> >> ACTIVEMQ_BASE: > >> >> > C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\.. > >> >> Loading message broker from: xbean:activemq.xml > >> >> INFO DefaultCamelContext - JMX enabled. Using > >> >> InstrumentationLifecycleStrategy. > >> >> INFO MLog - MLog clients using log4j > >> logging. > >> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2 > >> [built > >> >> 21-May-2007 15:04:56; debug? true; trace: 10] > >> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool... > >> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, > >> >> acquireRetryAttemp > >> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false, > >> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false, > >> >> checkoutTimeout - > >> >> > 30000, connectionCustomizerClassName -> null, > >> connectionTesterClassName > >> >> -> > >> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName = -> > >> >> 2rvy1 > >> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> > >> false, > >> >> description -> null, driverClass -> > >> >> com.microsoft.sqlserver.jdbc.SQLServerDri > >> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransaction= s > >> -> > >> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8, > >> >> idleConnectionTestPerio > >> >> d -> 30, initialPoolSize -> 10, jdbcUrl -> > >> >> jdbc:sqlserver://QG076619.qg.com:1433 > >> >> ;databaseName=3Dactivedb;user=3Dsa;password=3Dpa$$word, > >> >> maxAdministrativeTas > >> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30, > >> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatement= s > >> -> > >> >> 200, > >> >> maxStatementsPerCon > >> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3, > >> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> > >> null, > >> >> properties -> {user=3D** > >> >> ****, password=3D******}, propertyCycle -> 0, testConnectionOnCheck= in > -> > >> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeo= ut > >> -> > >> >> 0, > >> >> use > >> >> sTraditionalReflectiveProxies -> false ] > >> >> INFO BrokerService - Using Persistence Adapter: > >> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [ > >> >> acquireIncrement > >> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000, > >> >> autoCommitOnClose -> false, automaticTestTable -> con_test, > >> >> breakAfterAcquireFailure -> > >> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> > >> null, > >> >> connectionTesterClassName -> > >> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, > >> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8, > >> >> debugUnreturnedConnectionStackTraces -> false, description -> null, > >> >> driverClass -> com.microsoft.sql > >> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null, > >> >> forceIgnoreUnresolvedTransactions -> false, identityToken -> > >> >> 2rvy117z1wqpy2hxzkebp|18105e8, > >> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl -> > >> >> jdbc:sqlserver://QG076619.qg.com:1433 > >> >> ;databaseName=3Dactivedb;user=3Dsa;password=3Dpa$$wo > >> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, > maxIdleTime > >> -> > >> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, > >> maxStatements > >> >> -> > >> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10, > >> numHelperThreads > >> >> -> > >> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -= > > >> null > >> >> , properties -> {user=3D******, password=3D******}, propertyCycle -= > 0, > >> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false= , > >> >> unreturnedCon > >> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]) > >> >> INFO JDBCPersistenceAdapter - Database driver recognized: > >> >> [microsoft_sql_server_2005_jdbc_driver] > >> >> INFO DefaultDatabaseLocker - Attempting to acquire the > >> >> exclusive > >> >> lock to become the Master broker > >> >> INFO DefaultDatabaseLocker - Becoming the master on > >> dataSource: > >> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, > >> >> acquireRe > >> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> > >> >> false, > >> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false, > >> >> checkout > >> >> Timeout -> 30000, connectionCustomizerClassName -> null, > >> >> connectionTesterClassName -> > >> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName > >> >> -> 2rvy117z1wqpy2hxzkebp|18105e8, > >> debugUnreturnedConnectionStackTraces > >> >> -> > >> >> false, description -> null, driverClass -> > >> >> com.microsoft.sqlserver.jdbc.SQL > >> >> ServerDriver, factoryClassLocation -> null, > >> >> forceIgnoreUnresolvedTransactions -> false, identityToken -> > >> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection > >> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl -> > >> >> jdbc:sqlserver://QG076619.qg.com:1433 > >> >> ;databaseName=3Dactivedb;user=3Dsa;password=3Dpa$$word, > >> >> maxAdminist > >> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30, > >> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatement= s > >> -> > >> >> 200, > >> >> maxStateme > >> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3, > >> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> > >> null, > >> >> properties -> > >> >> {user=3D******, password=3D******}, propertyCycle -> 0, > >> >> testConnectionOnCheckin > >> >> -> false, testConnectionOnCheckout -> false, > >> unreturnedConnectionTimeout > >> >> -> 0, usesTraditionalReflectiveProxies -> false ] > >> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message > >> Broker > >> >> (master) is starting > >> >> INFO BrokerService - For help or more information > >> >> please > >> >> see: http://activemq.apache.org/ > >> >> INFO TransportServerThreadSupport - Listening for connections at= : > >> >> tcp://QG076619:61616 > >> >> INFO TransportConnector - Connector openwire Started > >> >> INFO TransportServerThreadSupport - Listening for connections at= : > >> >> ssl://QG076619:61617 > >> >> INFO TransportConnector - Connector ssl Started > >> >> INFO TransportServerThreadSupport - Listening for connections at= : > >> >> stomp://QG076619:61613 > >> >> INFO TransportConnector - Connector stomp Started > >> >> INFO TransportServerThreadSupport - Listening for connections at= : > >> >> xmpp://QG076619:61222 > >> >> INFO TransportConnector - Connector xmpp Started > >> >> INFO BrokerService - ActiveMQ JMS Message Broker > >> >> (master, > >> >> ID:QG076619-3632-1236812198396-0:0) started > >> >> INFO log - Logging to > >> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via > >> >> org.mortbay.log.Slf4jLog > >> >> INFO log - jetty-6.1.9 > >> >> INFO WebConsoleStarter - ActiveMQ WebConsole > >> initialized. > >> >> INFO /admin - Initializing Spring > >> >> FrameworkServlet > >> >> 'dispatcher' > >> >> INFO log - ActiveMQ Console at > >> >> http://0.0.0.0:8162/admin > >> >> INFO log - ActiveMQ Web Demos at > >> >> http://0.0.0.0:8162/demo > >> >> INFO log - RESTful file access > application > >> at > >> >> http://0.0.0.0:8162/fileserver > >> >> INFO log - Started > >> >> SelectChannelConnector@0.0.0.0:8162 > >> >> WARN BrokerRegistry - Broker localhost not started > so > >> >> using > >> >> master instead > >> >> INFO TransportConnector - Connector vm://localhost > >> Started > >> >> > >> >> > >> >> Then BOOM! Although behavior is different. Now it takes longer to > >> stop. > >> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to > >> >> shutdown." > >> >> If you look closely you can see activeMQ is timing out. > >> >> > >> >> > >> >> > >> >> > >> >> ERROR DefaultDatabaseLocker - Failed to update database > lock: > >> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset b= y > >> pe > >> >> et write error > >> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset b= y > >> >> peer: > >> >> socket write error > >> >> at > >> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown > >> >> Source) > >> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown > >> Source) > >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown > Source) > >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknow= n > >> >> Source) > >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown > >> >> Source) > >> >> at > >> com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown > >> >> Source) > >> >> at > >> >> > >> >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePrepared= Statement(Unknown > >> >> Source) > >> >> at > >> >> > >> >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.d= oExecute(Unknown > >> >> Source) > >> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown > >> Source) > >> >> at > >> >> > >> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknow= n > >> >> Source) > >> >> at > >> >> > com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown > >> >> Source) > >> >> at > >> >> > >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unkno= wn > >> >> Source) > >> >> at > >> >> > >> >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unk= nown > >> >> Source) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxy= PreparedStatement.java:105) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDat= abaseLocker.java:118) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAli= ve(JDBCPersistenceAdapter.java:481) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersisten= ceAdapter.java:183) > >> >> at > >> >> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441= ) > >> >> at > >> >> > >> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317= ) > >> >> at > >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acce= ss$101(ScheduledThreadPoolExecutor.java:98) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runP= eriodic(ScheduledThreadPoolExecutor.java:181) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(= ScheduledThreadPoolExecutor.java:205) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908) > >> >> at java.lang.Thread.run(Thread.java:619) > >> >> INFO GooGooStatementCache - Problem with checked-in > >> Statement, > >> >> discarding. > >> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is > >> >> closed. > >> >> at > >> >> > >> > com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unkno= wn > >> >> Source) > >> >> at > >> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknow= n > >> >> Source) > >> >> at > >> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown > >> >> Source) > >> >> at > >> >> > >> >> > >> > com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(U= nknown > >> >> Source) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStat= ementCache.java:604) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStat= ementCache.java:236) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledCo= nnection.java:282) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPrepared= Statement.java:1807) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDat= abaseLocker.java:127) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAli= ve(JDBCPersistenceAdapter.java:481) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersisten= ceAdapter.java:183) > >> >> at > >> >> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441= ) > >> >> at > >> >> > >> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317= ) > >> >> at > >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acce= ss$101(ScheduledThreadPoolExecutor.java:98) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runP= eriodic(ScheduledThreadPoolExecutor.java:181) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(= ScheduledThreadPoolExecutor.java:205) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886) > >> >> at > >> >> > >> >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908) > >> >> at java.lang.Thread.run(Thread.java:619) > >> >> INFO JDBCPersistenceAdapter - No longer able to keep the > >> >> exclusive > >> >> lock so giving up being a master > >> >> INFO BrokerService - ActiveMQ Message Broker > >> (master, > >> >> ID:QG076619-3332-1236811661934-0:0) is shutting down > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> WARN ThreadPoolAsynchronousRunner - > >> >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b0076= 6 > >> >> -- APPARENT DEADLOCK!!! Crea > >> >> rgency threads for unassigned pending tasks! > >> >> WARN ThreadPoolAsynchronousRunner - > >> >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b0076= 6 > >> >> -- APPARENT DEADLOCK!!! Comp > >> >> tus: > >> >> Managed Threads: 3 > >> >> Active Threads: 3 > >> >> Active Tasks: > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a > >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097 > >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49 > >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea > >> >> Pending Tasks: > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212= a > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087 > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561 > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961= b > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141 > >> >> > >> >> > >> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405= f6 > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945 > >> >> > >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3 > >> >> > >> >> > >> > com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc > >> >> Pool thread stack traces: > >> >> > >> >> > >> >> > >> > Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,= main] > >> >> java.lang.Thread.sleep(Native Method) > >> >> > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResour= cePool.java:1805) > >> >> > >> >> > >> >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPo= olAsynchronousRunner.java:547) > >> >> > >> >> > >> >> > >> > Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,= main] > >> >> java.lang.Thread.sleep(Native Method) > >> >> > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResour= cePool.java:1805) > >> >> > >> >> > >> >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPo= olAsynchronousRunner.java:547) > >> >> > >> >> > >> >> > >> > Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,= main] > >> >> java.lang.Thread.sleep(Native Method) > >> >> > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResour= cePool.java:1805) > >> >> > >> >> > >> >> > >> > com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPo= olAsynchronousRunner.java:547) > >> >> > >> >> > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> INFO TransportConnection - The connection to '/ > >> 10.0.1.5:3333' > >> >> is > >> >> taking a long time to shutdown. > >> >> ERROR QueueStorePrefetch - Failed to get message count > >> >> java.io.IOException: An attempt by a client to checkout a Connectio= n > >> has > >> >> timed out. > >> >> at > >> >> > >> >> > >> > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.jav= a:45) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(Transacti= onContext.java:61) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCou= nt(DefaultJDBCAdapter.java:687) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessa= geStore.java:208) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageS= tore.java:79) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize= (QueueStorePrefetch.java:63) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(Abstract= StoreCursor.java:212) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueC= ursor.java:257) > >> >> at > >> >> > >> > org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308= ) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractR= egion.java:324) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroke= r.java:392) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:= 114) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:= 114) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker= .java:222) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:= 114) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrok= erFilter.java:121) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConsumer(Tran= sportConnection.java:565) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveSession(Trans= portConnection.java:600) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConnection(Tr= ansportConnection.java:705) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection= .java:1000) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedT= ransportConnection.java:74) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.= java:925) > >> >> Caused by: java.sql.SQLException: An attempt by a client to checkou= t > a > >> >> Connection has timed out. > >> >> at > >> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106) > >> >> at > com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnectio= n(C3P0PooledConnectionPool.java:527) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abstr= actPoolBackedDataSource.java:128) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(Transacti= onContext.java:54) > >> >> ... 20 more > >> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client > >> timed > >> >> out > >> >> while waiting to acquire a resource from com.mchange.v2.resourcepo > >> >> ResourcePool@d964af -- timeout at awaitAvailable() > >> >> at > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourc= ePool.java:1317) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(Basi= cResourcePool.java:557) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResou= rcePool.java:477) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnectio= n(C3P0PooledConnectionPool.java:525) > >> >> ... 22 more > >> >> WARN TransportConnection - Failed to remove consumer: > >> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason: > >> >> java.lang.RuntimeException > >> >> o.IOException: An attempt by a client to checkout a Connection has > >> timed > >> >> out. > >> >> java.lang.RuntimeException: java.io.IOException: An attempt by a > >> client > >> >> to > >> >> checkout a Connection has timed out. > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize= (QueueStorePrefetch.java:66) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(Abstract= StoreCursor.java:212) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueC= ursor.java:257) > >> >> at > >> >> > >> > org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308= ) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractR= egion.java:324) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroke= r.java:392) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:= 114) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:= 114) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker= .java:222) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:= 114) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrok= erFilter.java:121) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConsumer(Tran= sportConnection.java:565) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveSession(Trans= portConnection.java:600) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.processRemoveConnection(Tr= ansportConnection.java:705) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection= .java:1000) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedT= ransportConnection.java:74) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.= java:925) > >> >> Caused by: java.io.IOException: An attempt by a client to checkout = a > >> >> Connection has timed out. > >> >> at > >> >> > >> >> > >> > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.jav= a:45) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(Transacti= onContext.java:61) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCou= nt(DefaultJDBCAdapter.java:687) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessa= geStore.java:208) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageS= tore.java:79) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize= (QueueStorePrefetch.java:63) > >> >> ... 16 more > >> >> Caused by: java.sql.SQLException: An attempt by a client to checkou= t > a > >> >> Connection has timed out. > >> >> at > >> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106) > >> >> at > com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnectio= n(C3P0PooledConnectionPool.java:527) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abstr= actPoolBackedDataSource.java:128) > >> >> at > >> >> > >> >> > >> > org.apache.activemq.store.jdbc.TransactionContext.getConnection(Transacti= onContext.java:54) > >> >> ... 20 more > >> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client > >> timed > >> >> out > >> >> while waiting to acquire a resource from com.mchange.v2.resourcepo > >> >> ResourcePool@d964af -- timeout at awaitAvailable() > >> >> at > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourc= ePool.java:1317) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(Basi= cResourcePool.java:557) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResou= rcePool.java:477) > >> >> at > >> >> > >> >> > >> > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnectio= n(C3P0PooledConnectionPool.java:525) > >> >> ... 22 more > >> >> INFO TransportConnector - Connector openwire Stopped > >> >> > >> >> > >> >> bsnyder wrote: > >> >> > > >> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings > >> >> wrote: > >> >> >> > >> >> >> Thanks for the response Bruce. :) > >> >> >> > >> >> >> To my knowledge we aren't using a connection pooler. We're just > >> >> >> configuring > >> >> >> the activemq xml and letting activeMQ do its thing. One of my > >> >> colleagues > >> >> >> mentioned SNAC since we are using Sql Server, not sure if that i= s > >> >> similar > >> >> >> to > >> >> >> a connection pooler or not. > >> >> > > >> >> > The job of a JDBC connection pooler is to keep a pool of JDBC > >> >> > connections available for use by a Java application. IIRC, SNAC > >> stands > >> >> > for SQL Native Client which has nothing to do with a JDBC > connection > >> >> > pooler. > >> >> > > >> >> >> In our scenario during a Sql Server failover it may take 15 > >> minutes. > >> >> We > >> >> >> aren't using a high availability cluster. Does a connection > pooler > >> >> give > >> >> >> the > >> >> >> capability of dealing with this so that ActiveMQ doesn't shutdow= n? > >> >> > > >> >> > One of the options in a good JDBC connection pooler is the abilit= y > >> to > >> >> > test JDBC connections as it hands them out to the Java applicatio= n. > >> If > >> >> > the connection is stale for any reason the JDBC connection pooler > >> will > >> >> > evict that connection from the pool and try the next one until it > >> >> > reaches one that is alive. To my knowledge, some JDBC connection > >> >> > poolers provide a the ability to configure some attributes like > >> number > >> >> > of retries, etc. E.g., here is some information about what c3p0 > >> >> > offers: > >> >> > > >> >> > > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery > >> >> > > >> >> > The one big caveat I will mention is that longer outage periods > such > >> >> > as the 15 minute outage you mention are not typically handled by = a > >> >> > JDBC connection pooler as they're mainly geared toward handling > >> >> > momentary outages. As an example, notice that the default value f= or > >> >> > acquireRetryDelay in c3p0 is only one second: > >> >> > > >> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay > >> >> > > >> >> > But you could certainly configure this to any value you want. I > >> >> > encourage you to test this in your environment before deploying t= o > >> >> > production. > >> >> > > >> >> > Bruce > >> >> > -- > >> >> > perl -e 'print > >> >> > > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=3D6-E+G-N>61E >> >> > );' > >> >> > > >> >> > Apache ActiveMQ - http://activemq.apache.org/ > >> >> > Apache Camel - http://camel.apache.org/ > >> >> > Apache ServiceMix - http://servicemix.apache.org/ > >> >> > > >> >> > Blog: http://bruceblog.org/ > >> >> > > >> >> > > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> > http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-= goes-down--tp22440338p22466448.html > >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> >> > >> >> > >> > > >> > > >> > -- > >> > http://blog.garytully.com > >> > > >> > Open Source SOA > >> > http://FUSESource.com > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-= goes-down--tp22440338p22939332.html > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > > http://blog.garytully.com > > > > Open Source SOA > > http://FUSESource.com > > > > > > -- > View this message in context: > http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-= goes-down--tp22440338p22973315.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > --=20 http://blog.garytully.com Open Source SOA http://FUSESource.com --00163662e5695e4ea40467212950--