Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Fri, 8 Apr 2011 06:01:08 +0000 (UTC)
From: "Oleg Kiorsak (JIRA)" <jira@apache.org>
To: dev@activemq.apache.org
Message-ID: 
 <1995212532.43065.1302242468907.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (AMQ-1780) ActiveMQ broker does not
 automatically reconnect if the connection to the database is lost
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/AMQ-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017299#comment-13017299 ] 

Oleg Kiorsak commented on AMQ-1780:
-----------------------------------

Hi Gary 

interestingly, on some of the attempts I did today all transport connectors do stop all at same time (when DB conn is lost)...but in either case however they are not been restarted automatically when DB restored...

I tried to produce a thread dump - this is what could fit on shell screen:

---
"qtp768129156-32" prio=3 tid=0x0000000101885800 nid=0x23 waiting on condition [0xfffffffd6bafe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffedce04c38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
        at java.lang.Thread.run(Thread.java:619)

"qtp768129156-31" prio=3 tid=0x00000001010dc000 nid=0x22 waiting on condition [0xfffffffd6bcfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffedce04c38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
        at java.lang.Thread.run(Thread.java:619)

"qtp768129156-30" prio=3 tid=0x00000001013fe000 nid=0x21 waiting on condition [0xfffffffd6befe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffedce04c38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
        at java.lang.Thread.run(Thread.java:619)

"qtp768129156-29" prio=3 tid=0x00000001013fd800 nid=0x20 waiting on condition [0xfffffffd6c0fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffedce04c38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
        at java.lang.Thread.run(Thread.java:619)

"qtp768129156-28 - Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=3 tid=0x0000000100e89000 nid=0x1f runnable [0xfffffffd6c2fe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0xffffffff03259268> (a sun.nio.ch.Util$1)
        - locked <0xffffffff03259250> (a java.util.Collections$UnmodifiableSet)
        - locked <0xffffffff03258ef0> (a sun.nio.ch.DevPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:480)
        at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:134)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:793)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Broker[MobTechTest1] Scheduler" daemon prio=3 tid=0x000000010109d000 nid=0x18 in Object.wait() [0xfffffffd6d6ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xfffffffedc814190> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xfffffffedc814190> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"ActiveMQ Cleanup Timer" daemon prio=3 tid=0x0000000100e35000 nid=0x17 waiting on condition [0xfffffffd6e8fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffedc766668> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Connection(5)-10.38.58.172" daemon prio=3 tid=0x0000000100f54800 nid=0x16 runnable [0xfffffffd6d8fe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0xfffffffed9747f10> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"RMI RenewClean-[10.66.179.1:50344]" daemon prio=3 tid=0x0000000100efd800 nid=0x15 in Object.wait() [0xfffffffd6dafe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xfffffffed7bf5598> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xfffffffed7bf5598> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

"RMI Scheduler(0)" daemon prio=3 tid=0x0000000100dc9800 nid=0x14 waiting on condition [0xfffffffd6defe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffd780eb878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"GC Daemon" daemon prio=3 tid=0x000000010126f800 nid=0x12 in Object.wait() [0xfffffffd6e2ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xfffffffd780001d8> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0xfffffffd780001d8> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=3 tid=0x00000001010f9000 nid=0x11 in Object.wait() [0xfffffffd6e4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xfffffffd78000290> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xfffffffd78000290> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=3 tid=0x0000000101009800 nid=0x10 runnable [0xfffffffd6e6fe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xfffffffd782d55d0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-2011" daemon prio=3 tid=0x0000000101139800 nid=0xe runnable [0xfffffffd6eafe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xfffffffd780007b0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=3 tid=0x00000001007ba800 nid=0xc runnable [0xfffffffd6eefe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xfffffffd7807dcc8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=3 tid=0x00000001004fe000 nid=0xb runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=3 tid=0x00000001004f7800 nid=0xa waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=3 tid=0x00000001004f4800 nid=0x9 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=3 tid=0x00000001004f2800 nid=0x8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=3 tid=0x00000001004d2800 nid=0x7 in Object.wait() [0xfffffffd704ff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xfffffffd7807e7d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xfffffffd7807e7d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=3 tid=0x00000001004cb800 nid=0x6 in Object.wait() [0xfffffffd706ff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xfffffffd780001b8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0xfffffffd780001b8> (a java.lang.ref.Reference$Lock)

"main" prio=3 tid=0x0000000100114800 nid=0x2 in Object.wait() [0xffffffff7c4fd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xffffffff03266308> (a [Z)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.console.command.StartCommand.waitForShutdown(StartCommand.java:161)
        - locked <0xffffffff03266308> (a [Z)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:104)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
        at org.apache.activemq.console.Main.main(Main.java:107)

"VM Thread" prio=3 tid=0x00000001004c7800 nid=0x5 runnable

"GC task thread#0 (ParallelGC)" prio=3 tid=0x000000010011c000 nid=0x3 runnable

"GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100128000 nid=0x4 runnable

"VM Periodic Task Thread" prio=3 tid=0x000000010066c000 nid=0xd waiting on condition

JNI global references: 767

Heap
 PSYoungGen      total 2447872K, used 1608619K [0xfffffffecd800000, 0xffffffff78400000, 0xffffffff78400000)
  eden space 2098176K, 76% used [0xfffffffecd800000,0xffffffff2faeaca0,0xffffffff4d900000)
  from space 349696K, 0% used [0xffffffff4d900000,0xffffffff4d900000,0xffffffff62e80000)
  to   space 349696K, 0% used [0xffffffff62e80000,0xffffffff62e80000,0xffffffff78400000)
 PSOldGen        total 5595136K, used 4074K [0xfffffffd78000000, 0xfffffffecd800000, 0xfffffffecd800000)
  object space 5595136K, 0% used [0xfffffffd78000000,0xfffffffd783fa8a8,0xfffffffecd800000)
 PSPermGen       total 45056K, used 44664K [0xfffffffd72c00000, 0xfffffffd75800000, 0xfffffffd78000000)
  object space 45056K, 99% used [0xfffffffd72c00000,0xfffffffd7579e1b8,0xfffffffd75800000)


[mdtapdot01]#
[mdtapdot01]#  INFO | waiting for broker persistence adapter checkpoint to succeed before restarting transports
 INFO | waiting for broker persistence adapter checkpoint to succeed before restarting transports
 INFO | waiting for broker persistence adapter checkpoint to succeed before restarting transports


> ActiveMQ broker does not automatically reconnect if the connection to the database is lost
> ------------------------------------------------------------------------------------------
>
>                 Key: AMQ-1780
>                 URL: https://issues.apache.org/jira/browse/AMQ-1780
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.0.0
>         Environment: Windows 2003 Server
>            Reporter: Jaya Srinivasan
>            Assignee: Gary Tully
>             Fix For: 5.5.0
>
>
> hi
> We are noticing that after any SQL Server restart or network blip between ActiveMQ and the database, after the connection or the database comes back online activeMQ broker needs to be restarted as well i.e it doesn't automatically re-establish connection to the database as result any message send fails because the broker is still using the stale connection to the database. 
> Is this designed behaviour or a bug? we are using ActiveMQ 5.0.0 and the latest version of the JSQLConnect database driver: version 5.7. The database we are using is MS SQL Server 2005 
> Right now, in our production environment any time we have network maintenance or database restart we also have to restart the ActiveMQ broker which is not a good option for us. 
> Also, We are using a single ActiveMQ broker and not the JDBC(Master/Slave) set up.
> Issue details in
> http://www.nabble.com/Database-connection-between-ActiveMQ-and-broker-td17321330s2354.html
> Please let me know if I need to give more information
> thanks
> jaya

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira