activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Schöchlin (JIRA) <j...@apache.org>
Subject [jira] Issue Comment Edited: (AMQ-2680) Producer Flow Control seems to be broken
Date Wed, 07 Apr 2010 10:57:09 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58694#action_58694 ] 

Marc Schöchlin edited comment on AMQ-2680 at 4/7/10 10:55 AM:
--------------------------------------------------------------

Thanks for the configuration hints.
 
It seems now working as expected.
I strongly recommend to put some logic in the configuration handling to detect bda configurations like that - sysadmins do normally not know
details about the internals of the activemq message store implementation.

Nevertheless it seems that there is still a bug - after some time my messagequeue stopped receiving messages:
(mention the configuration attached as activemq-broken-flow-control.xml)

- Shutdown of producer was not possible
- Shutdown of broker was not possible (kill -KILL was the only solution)
- After restart the blocked queue does still not have any pending messages, but the queue is still
  blocked  by producer flow control
  (created a threaddump by kill -QUIT, see beyond)

Threadump;:
{code}
ACTIVEMQ_HOME: /srv/activemq/install/current
ACTIVEMQ_BASE: /srv/activemq/install/current
Loading message broker from: xbean:activemq.xml
 INFO | Using Persistence Adapter: AMQPersistenceAdapter(/srv/activemq/work/data)
 INFO | AMQStore starting using directory: /srv/activemq/work/data
 INFO | Kaha Store using data directory /srv/activemq/work/data/kr-store/state
 INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:51109/jmxrmi
 INFO | Active data files: [1, 2, 3]
 INFO | ActiveMQ 5.3.1 JMS Message Broker (localhost) is starting
 INFO | For help or more information please see: http://activemq.apache.org/
 INFO | Kaha Store using data directory /srv/activemq/work/data/kr-store/data
 INFO | Listening for connections at: tcp://esxschnapp3-x3850.argl.iv.FOOBAR.de:61616
 INFO | Connector openwire Started
 INFO | ActiveMQ JMS Message Broker (localhost, ID:esxschnapp3-x3850.argl.iv.FOOBAR.de-48924-1270635425111-0:0) started
 INFO | Logging to org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
 INFO | jetty-6.1.9
 INFO | ActiveMQ WebConsole initialized.
 INFO | Initializing Spring FrameworkServlet 'dispatcher'
 INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
 INFO | Initializing Spring root WebApplicationContext
 INFO | Successfully connected to tcp://localhost:61616
 INFO | Camel Console at http://0.0.0.0:8161/camel
 INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
 INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
 INFO | Started SelectChannelConnector@0.0.0.0:8161
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-9828-1270572546517-0:17247:1:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 30s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 30s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-9828-1270572546517-0:17247:1:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 60s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 60s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 90s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 30s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 120s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 60s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 150s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 90s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 180s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 120s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 210s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 150s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 240s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 180s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 270s)
2010-04-07 12:21:55
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):

"InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@717fed4" daemon prio=10 tid=0x00007f71b0a1e000 nid=0x4164 waiting on condition [0x00007f71b4a3e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c83a2210> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
        at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@6cfcaac5" daemon prio=10 tid=0x00007f71b0a1d000 nid=0x4163 waiting on condition [0x00007f71af7f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c83a2210> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
        at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"JMX server connection timeout 94" daemon prio=10 tid=0x0000000040530000 nid=0x414e in Object.wait() [0x00007f71b4c40000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd665e8> (a [I)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
        - locked <0x00007f71efd665e8> (a [I)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Connection(4)-10.130.39.230" daemon prio=10 tid=0x00000000409e8800 nid=0x414d in Object.wait() [0x00007f71af3d6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efbd6710> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
        - locked <0x00007f71efbd6710> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
        at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Connection(7)-10.130.39.230" daemon prio=10 tid=0x0000000040479800 nid=0x414c runnable [0x00007f71b6760000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        - locked <0x00007f71efd8af60> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"btpool0-8" prio=10 tid=0x00007f71a8b72800 nid=0x4143 in Object.wait() [0x00007f71af5f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc85688> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc85688> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"ActiveMQ Transport: tcp:///10.73.11.1:46264" daemon prio=10 tid=0x00000000402f7000 nid=0x4142 in Object.wait() [0x00007f71aefd4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:100)
        - locked <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.waitForSpace(Queue.java:1726)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:568)
        - locked <0x00007f71efb09768> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:539)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:446)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:216)
        - locked <0x00007f71efc544c0> (a org.apache.activemq.transport.InactivityMonitor$1)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Connection Dispatcher: /10.73.11.1:46264" daemon prio=10 tid=0x0000000040236000 nid=0x4141 in Object.wait() [0x00007f71af0d5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc29000> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efc29000> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"btpool0-6" prio=10 tid=0x00007f71a8449000 nid=0x4138 in Object.wait() [0x00007f71af4f3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc86118> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc86118> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-4" prio=10 tid=0x00007f71a86dd800 nid=0x4136 in Object.wait() [0x00007f71af6f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc84bf8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc84bf8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-2" prio=10 tid=0x0000000040a17800 nid=0x4134 in Object.wait() [0x00007f71af8f7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc80150> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc80150> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"Timer-3" daemon prio=10 tid=0x00007f71b056c000 nid=0x4132 in Object.wait() [0x00007f71afaf9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd143d8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efd143d8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"Timer-2" daemon prio=10 tid=0x00007f71b0103000 nid=0x4131 in Object.wait() [0x00007f71afbfa000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd67d68> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efd67d68> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"ActiveMQ Session: ID:esxschnapp3-x3850.argl.iv.FOOBAR.de-48924-1270635425111-2:0:1" prio=10 tid=0x00007f71a8034000 nid=0x4130 in Object.wait() [0x00007f71afcfb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd09b68> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efd09b68> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Topic  ActiveMQ.Advisory.Consumer.Queue.example.A" daemon prio=10 tid=0x00000000409d8000 nid=0x412f in Object.wait() [0x00007f71afdfc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb4cc78> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb4cc78> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"DefaultMessageListenerContainer-1" prio=10 tid=0x00000000409dc000 nid=0x412e in Object.wait() [0x00007f71b4114000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd09d90> (a java.lang.Object)
        at org.apache.activemq.MessageDispatchChannel.dequeue(MessageDispatchChannel.java:77)
        - locked <0x00007f71efd09d90> (a java.lang.Object)
        at org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:447)
        at org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:572)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:405)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Transport: tcp:///127.0.0.1:52732" daemon prio=10 tid=0x00000000403ab800 nid=0x412d runnable [0x00007f71afefd000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
        at java.io.DataInputStream.readInt(Unknown Source)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Connection Dispatcher: /127.0.0.1:52732" daemon prio=10 tid=0x00000000403a9800 nid=0x412c in Object.wait() [0x00007f71afffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71ef966be8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71ef966be8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=10 tid=0x0000000040947000 nid=0x412a runnable [0x00007f71b4215000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
        at java.io.DataInputStream.readInt(Unknown Source)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Failover Worker: 1811450195" daemon prio=10 tid=0x00007f71a8009800 nid=0x4129 in Object.wait() [0x00007f71b4316000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd11618> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efd11618> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Timer-1" daemon prio=10 tid=0x00007f71b0780800 nid=0x411e in Object.wait() [0x00007f71b4417000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c7fdd550> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71c7fdd550> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"Timer-0" daemon prio=10 tid=0x00007f71b0018800 nid=0x411d in Object.wait() [0x00007f71b4539000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc86ea8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efc86ea8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"Topic  ActiveMQ.Advisory.Producer.Queue.queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x0000000040827000 nid=0x411c in Object.wait() [0x00007f71b463a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb4ad20> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb4ad20> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"btpool0-0 - Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10 tid=0x00007f71b0137800 nid=0x411b runnable [0x00007f71b4d41000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
        at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00007f71efd0c458> (a sun.nio.ch.Util$1)
        - locked <0x00007f71efd0c440> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00007f71efbda070> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:406)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:537)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

"Topic  ActiveMQ.Advisory.Connection" daemon prio=10 tid=0x000000004018a000 nid=0x411a in Object.wait() [0x00007f71b4e42000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb4ea18> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb4ea18> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"InactivityMonitor WriteCheck" daemon prio=10 tid=0x0000000040829000 nid=0x4119 in Object.wait() [0x00007f71b473b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd31248> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efd31248> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"InactivityMonitor ReadCheck" daemon prio=10 tid=0x0000000040838800 nid=0x4118 in Object.wait() [0x00007f71b483c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc87c68> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efc87c68> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"ActiveMQ Transport: tcp:///10.73.11.1:46253" daemon prio=10 tid=0x0000000040824800 nid=0x4117 in Object.wait() [0x00007f71b493c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:100)
        - locked <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.waitForSpace(Queue.java:1726)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:568)
        - locked <0x00007f71c3d75ff8> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:539)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:446)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:216)
        - locked <0x00007f71efad4c00> (a org.apache.activemq.transport.InactivityMonitor$1)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Connection Dispatcher: /10.73.11.1:46253" daemon prio=10 tid=0x00000000406ee000 nid=0x4115 in Object.wait() [0x00007f71b4b3f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efab86f8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efab86f8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"ActiveMQ Transport Server: tcp://0.0.0.0:61616" daemon prio=10 tid=0x00007f71b0456800 nid=0x4111 runnable [0x00007f71b4f43000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71efc552b8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:280)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616" daemon prio=10 tid=0x00007f71b0457800 nid=0x4110 waiting on condition [0x00007f71b5044000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efc555a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:352)
        at java.lang.Thread.run(Unknown Source)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b00c2000 nid=0x410f waiting on condition [0x00007f71b514c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efb0bcb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:DLQ.queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b00c0000 nid=0x410e in Object.wait() [0x00007f71b524d000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb0b1e8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb0b1e8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://DLQ.queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b00be800 nid=0x410d in Object.wait() [0x00007f71b534e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb0a5e8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb0a5e8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b00bc000 nid=0x410c waiting on condition [0x00007f71b544f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efad75c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:example.A" daemon prio=10 tid=0x00007f71b04c5000 nid=0x410b in Object.wait() [0x00007f71b5550000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efad6588> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efad6588> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://example.A" daemon prio=10 tid=0x00007f71b04c3000 nid=0x410a in Object.wait() [0x00007f71b5651000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efad59d8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efad59d8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b04c8000 nid=0x4109 waiting on condition [0x00007f71b5752000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efb09a10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:queue/Einarbeitungsdaten_RemoteQ" daemon prio=10 tid=0x00007f71b04c7000 nid=0x4108 in Object.wait() [0x00007f71b5853000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb08f40> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb08f40> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://queue/Einarbeitungsdaten_RemoteQ" daemon prio=10 tid=0x00007f71b04c6000 nid=0x4107 in Object.wait() [0x00007f71b5954000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb08330> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb08330> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Topic  ActiveMQ.Advisory.Queue" daemon prio=10 tid=0x00007f71b0613800 nid=0x4106 in Object.wait() [0x00007f71b5a55000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb487a8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb487a8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b04c9800 nid=0x4105 waiting on condition [0x00007f71b5b56000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c3cbded0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b061e800 nid=0x4104 in Object.wait() [0x00007f71b5c57000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efac23b8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efac23b8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b04b3800 nid=0x4103 in Object.wait() [0x00007f71b5d58000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efac1990> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efac1990> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"ActiveMQ Journal Checkpoint Worker" daemon prio=10 tid=0x00007f71b05d0800 nid=0x4102 in Object.wait() [0x00007f71b6b72000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71ef960c48> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71ef960c48> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"RMI RenewClean-[10.130.18.244:35003]" daemon prio=10 tid=0x00007f71b04d8000 nid=0x4100 in Object.wait() [0x00007f71b5f5a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc91e50> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00007f71efc91e50> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Data File Writer" daemon prio=10 tid=0x00007f71b83b6000 nid=0x40ff in Object.wait() [0x00007f71b655e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c7fd9fa0> (a org.apache.activemq.kaha.impl.async.DataFileAppender$1)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.kaha.impl.async.NIODataFileAppender.processQueue(NIODataFileAppender.java:80)
        - locked <0x00007f71c7fd9fa0> (a org.apache.activemq.kaha.impl.async.DataFileAppender$1)
        at org.apache.activemq.kaha.impl.async.DataFileAppender$2.run(DataFileAppender.java:230)

"RMI Scheduler(0)" daemon prio=10 tid=0x00007f71b83b7000 nid=0x40fe waiting on condition [0x00007f71b665f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c28fd860> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"GC Daemon" daemon prio=10 tid=0x0000000040564800 nid=0x40fc in Object.wait() [0x00007f71b6970000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2926ee0> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(Unknown Source)
        - locked <0x00007f71c2926ee0> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=10 tid=0x000000004023a000 nid=0x40fb in Object.wait() [0x00007f71b6a71000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c28fbee0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00007f71c28fbee0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at sun.rmi.transport.ObjectTable$Reaper.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Accept-51109" daemon prio=10 tid=0x00007f71b8333800 nid=0x40f9 runnable [0x00007f71b6c73000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c28817a8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Scheduler" daemon prio=10 tid=0x00007f71b836e800 nid=0x40f8 in Object.wait() [0x00007f71b6d74000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c28ba498> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71c28ba498> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f71b0092000 nid=0x40f6 runnable [0x00007f71b70c6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c2881d38> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Accept-11099" daemon prio=10 tid=0x00007f71b0090000 nid=0x40f5 runnable [0x00007f71b71c7000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c28821c0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f71b0082000 nid=0x40f4 runnable [0x00007f71b72c8000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c2882630> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"CommunicatorServer" daemon prio=10 tid=0x00007f71b003c000 nid=0x40f3 runnable [0x00007f71b73c9000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked <0x00007f71c289e760> (a java.net.PlainDatagramSocketImpl)
        at java.net.PlainDatagramSocketImpl.receive(Unknown Source)
        - locked <0x00007f71c289e760> (a java.net.PlainDatagramSocketImpl)
        at java.net.DatagramSocket.receive(Unknown Source)
        - locked <0x00007f71c289e7d0> (a java.net.DatagramPacket)
        - locked <0x00007f71c289e718> (a java.net.DatagramSocket)
        at com.sun.jmx.snmp.daemon.SnmpAdaptorServer.doReceive(Unknown Source)
        at com.sun.jmx.snmp.daemon.CommunicatorServer.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Low Memory Detector" daemon prio=10 tid=0x00007f71b0002000 nid=0x40ef runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00007f71b826c000 nid=0x40ee waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00007f71b826a000 nid=0x40ed waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" daemon prio=10 tid=0x00007f71b8267800 nid=0x40ec runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x00007f71b8264000 nid=0x40eb runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f71b8258000 nid=0x40ea waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f71b823a000 nid=0x40e9 in Object.wait() [0x00007f71b7ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2927370> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00007f71c2927370> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x00007f71b8238000 nid=0x40e8 in Object.wait() [0x00007f71bc1af000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2926ed0> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x00007f71c2926ed0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x000000004011d800 nid=0x40d8 in Object.wait() [0x00007f71f6488000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd33780> (a [Z)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.console.command.StartCommand.waitForShutdown(StartCommand.java:161)
        - locked <0x00007f71efd33780> (a [Z)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:104)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
        at org.apache.activemq.console.Main.main(Main.java:107)

"VM Thread" prio=10 tid=0x00007f71b8233800 nid=0x40e7 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000040127800 nid=0x40da runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040129800 nid=0x40db runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000004012b800 nid=0x40dc runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x000000004012d000 nid=0x40dd runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x000000004012f000 nid=0x40de runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040131000 nid=0x40df runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000040132800 nid=0x40e0 runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000040134800 nid=0x40e1 runnable 

"GC task thread#8 (ParallelGC)" prio=10 tid=0x0000000040136800 nid=0x40e2 runnable 

"GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040138000 nid=0x40e3 runnable 

"GC task thread#10 (ParallelGC)" prio=10 tid=0x000000004013a000 nid=0x40e4 runnable 

"GC task thread#11 (ParallelGC)" prio=10 tid=0x000000004013c000 nid=0x40e5 runnable 

"GC task thread#12 (ParallelGC)" prio=10 tid=0x000000004013d800 nid=0x40e6 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007f71b0094800 nid=0x40f7 waiting on condition 

JNI global references: 6239

Heap
 PSYoungGen      total 199552K, used 51797K [0x00007f71e2800000, 0x00007f71f1f10000, 0x00007f71f2800000)
  eden space 158528K, 23% used [0x00007f71e2800000,0x00007f71e4bdcd18,0x00007f71ec2d0000)
  from space 41024K, 36% used [0x00007f71ef700000,0x00007f71f05b8820,0x00007f71f1f10000)
  to   space 47232K, 0% used [0x00007f71ec2d0000,0x00007f71ec2d0000,0x00007f71ef0f0000)
 PSOldGen        total 199616K, used 95454K [0x00007f71c2800000, 0x00007f71ceaf0000, 0x00007f71e2800000)
  object space 199616K, 47% used [0x00007f71c2800000,0x00007f71c8537b48,0x00007f71ceaf0000)
 PSPermGen       total 43264K, used 43152K [0x00007f71bd400000, 0x00007f71bfe40000, 0x00007f71c2800000)
  object space 43264K, 99% used [0x00007f71bd400000,0x00007f71bfe24138,0x00007f71bfe40000)

 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 210s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 300s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent floodi
{code} 


      was (Author: scoopex):
    Thanks for the configuration hints.
 
It seems now working as expected.
I strongly recommend to put some logic in the configuration handling to detect bda configurations like that - sysadmins do normally not know
details about the internals of the activemq message store implementation.

Nevertheless it seems that there is still a bug - after some time my messagequeue stopped receiving messages:
(mention the configuration attached as activemq-broken-flow-control.xml)

- Shutdown of producer was not possible
- Shutdown of broker was not possible (kill -KILL was the only solution)
- After restart the blocked queue does still not have any pending messages, but the queue is still
  blocked  by producer flow control
  (created a threaddump by kill -QUIT, see beyond)

Threadump;:
{code}
ACTIVEMQ_HOME: /srv/activemq/install/current
ACTIVEMQ_BASE: /srv/activemq/install/current
Loading message broker from: xbean:activemq.xml
 INFO | Using Persistence Adapter: AMQPersistenceAdapter(/srv/activemq/work/data)
 INFO | AMQStore starting using directory: /srv/activemq/work/data
 INFO | Kaha Store using data directory /srv/activemq/work/data/kr-store/state
 INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:51109/jmxrmi
 INFO | Active data files: [1, 2, 3]
 INFO | ActiveMQ 5.3.1 JMS Message Broker (localhost) is starting
 INFO | For help or more information please see: http://activemq.apache.org/
 INFO | Kaha Store using data directory /srv/activemq/work/data/kr-store/data
 INFO | Listening for connections at: tcp://esxschnapp3-x3850.argl.iv.FOOBAR.de:61616
 INFO | Connector openwire Started
 INFO | ActiveMQ JMS Message Broker (localhost, ID:esxschnapp3-x3850.argl.iv.FOOBAR.de-48924-1270635425111-0:0) started
 INFO | Logging to org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
 INFO | jetty-6.1.9
 INFO | ActiveMQ WebConsole initialized.
 INFO | Initializing Spring FrameworkServlet 'dispatcher'
 INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
 INFO | Initializing Spring root WebApplicationContext
 INFO | Successfully connected to tcp://localhost:61616
 INFO | Camel Console at http://0.0.0.0:8161/camel
 INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
 INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
 INFO | Started SelectChannelConnector@0.0.0.0:8161
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-9828-1270572546517-0:17247:1:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 30s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 30s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-9828-1270572546517-0:17247:1:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 60s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 60s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 90s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 30s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 120s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 60s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 150s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 90s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 180s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 120s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 210s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 150s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 240s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 180s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 270s)
2010-04-07 12:21:55
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):

"InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@717fed4" daemon prio=10 tid=0x00007f71b0a1e000 nid=0x4164 waiting on condition [0x00007f71b4a3e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c83a2210> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
        at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@6cfcaac5" daemon prio=10 tid=0x00007f71b0a1d000 nid=0x4163 waiting on condition [0x00007f71af7f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c83a2210> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
        at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"JMX server connection timeout 94" daemon prio=10 tid=0x0000000040530000 nid=0x414e in Object.wait() [0x00007f71b4c40000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd665e8> (a [I)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
        - locked <0x00007f71efd665e8> (a [I)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Connection(4)-10.130.39.230" daemon prio=10 tid=0x00000000409e8800 nid=0x414d in Object.wait() [0x00007f71af3d6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efbd6710> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
        - locked <0x00007f71efbd6710> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
        at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Connection(7)-10.130.39.230" daemon prio=10 tid=0x0000000040479800 nid=0x414c runnable [0x00007f71b6760000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        - locked <0x00007f71efd8af60> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"btpool0-8" prio=10 tid=0x00007f71a8b72800 nid=0x4143 in Object.wait() [0x00007f71af5f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc85688> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc85688> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"ActiveMQ Transport: tcp:///10.72.10.2:46264" daemon prio=10 tid=0x00000000402f7000 nid=0x4142 in Object.wait() [0x00007f71aefd4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:100)
        - locked <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.waitForSpace(Queue.java:1726)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:568)
        - locked <0x00007f71efb09768> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:539)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:446)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:216)
        - locked <0x00007f71efc544c0> (a org.apache.activemq.transport.InactivityMonitor$1)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Connection Dispatcher: /10.72.10.2:46264" daemon prio=10 tid=0x0000000040236000 nid=0x4141 in Object.wait() [0x00007f71af0d5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc29000> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efc29000> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"btpool0-6" prio=10 tid=0x00007f71a8449000 nid=0x4138 in Object.wait() [0x00007f71af4f3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc86118> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc86118> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-4" prio=10 tid=0x00007f71a86dd800 nid=0x4136 in Object.wait() [0x00007f71af6f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc84bf8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc84bf8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-2" prio=10 tid=0x0000000040a17800 nid=0x4134 in Object.wait() [0x00007f71af8f7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc80150> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
        - locked <0x00007f71efc80150> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"Timer-3" daemon prio=10 tid=0x00007f71b056c000 nid=0x4132 in Object.wait() [0x00007f71afaf9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd143d8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efd143d8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"Timer-2" daemon prio=10 tid=0x00007f71b0103000 nid=0x4131 in Object.wait() [0x00007f71afbfa000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd67d68> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efd67d68> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"ActiveMQ Session: ID:esxschnapp3-x3850.argl.iv.FOOBAR.de-48924-1270635425111-2:0:1" prio=10 tid=0x00007f71a8034000 nid=0x4130 in Object.wait() [0x00007f71afcfb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd09b68> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efd09b68> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Topic  ActiveMQ.Advisory.Consumer.Queue.example.A" daemon prio=10 tid=0x00000000409d8000 nid=0x412f in Object.wait() [0x00007f71afdfc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb4cc78> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb4cc78> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"DefaultMessageListenerContainer-1" prio=10 tid=0x00000000409dc000 nid=0x412e in Object.wait() [0x00007f71b4114000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd09d90> (a java.lang.Object)
        at org.apache.activemq.MessageDispatchChannel.dequeue(MessageDispatchChannel.java:77)
        - locked <0x00007f71efd09d90> (a java.lang.Object)
        at org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:447)
        at org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:572)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:405)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Transport: tcp:///127.0.0.1:52732" daemon prio=10 tid=0x00000000403ab800 nid=0x412d runnable [0x00007f71afefd000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
        at java.io.DataInputStream.readInt(Unknown Source)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Connection Dispatcher: /127.0.0.1:52732" daemon prio=10 tid=0x00000000403a9800 nid=0x412c in Object.wait() [0x00007f71afffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71ef966be8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71ef966be8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=10 tid=0x0000000040947000 nid=0x412a runnable [0x00007f71b4215000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
        at java.io.DataInputStream.readInt(Unknown Source)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Failover Worker: 1811450195" daemon prio=10 tid=0x00007f71a8009800 nid=0x4129 in Object.wait() [0x00007f71b4316000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd11618> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efd11618> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Timer-1" daemon prio=10 tid=0x00007f71b0780800 nid=0x411e in Object.wait() [0x00007f71b4417000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c7fdd550> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71c7fdd550> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"Timer-0" daemon prio=10 tid=0x00007f71b0018800 nid=0x411d in Object.wait() [0x00007f71b4539000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc86ea8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efc86ea8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"Topic  ActiveMQ.Advisory.Producer.Queue.queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x0000000040827000 nid=0x411c in Object.wait() [0x00007f71b463a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb4ad20> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb4ad20> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"btpool0-0 - Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10 tid=0x00007f71b0137800 nid=0x411b runnable [0x00007f71b4d41000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
        at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00007f71efd0c458> (a sun.nio.ch.Util$1)
        - locked <0x00007f71efd0c440> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00007f71efbda070> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:406)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:537)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

"Topic  ActiveMQ.Advisory.Connection" daemon prio=10 tid=0x000000004018a000 nid=0x411a in Object.wait() [0x00007f71b4e42000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb4ea18> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb4ea18> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"InactivityMonitor WriteCheck" daemon prio=10 tid=0x0000000040829000 nid=0x4119 in Object.wait() [0x00007f71b473b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd31248> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efd31248> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"InactivityMonitor ReadCheck" daemon prio=10 tid=0x0000000040838800 nid=0x4118 in Object.wait() [0x00007f71b483c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc87c68> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71efc87c68> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"ActiveMQ Transport: tcp:///10.72.10.2:46253" daemon prio=10 tid=0x0000000040824800 nid=0x4117 in Object.wait() [0x00007f71b493c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:100)
        - locked <0x00007f71c2957450> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.waitForSpace(Queue.java:1726)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:568)
        - locked <0x00007f71c3d75ff8> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:539)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:446)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:631)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:216)
        - locked <0x00007f71efad4c00> (a org.apache.activemq.transport.InactivityMonitor$1)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Connection Dispatcher: /10.72.10.2:46253" daemon prio=10 tid=0x00000000406ee000 nid=0x4115 in Object.wait() [0x00007f71b4b3f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efab86f8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efab86f8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"ActiveMQ Transport Server: tcp://0.0.0.0:61616" daemon prio=10 tid=0x00007f71b0456800 nid=0x4111 runnable [0x00007f71b4f43000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71efc552b8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:280)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616" daemon prio=10 tid=0x00007f71b0457800 nid=0x4110 waiting on condition [0x00007f71b5044000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efc555a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
        at org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:352)
        at java.lang.Thread.run(Unknown Source)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b00c2000 nid=0x410f waiting on condition [0x00007f71b514c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efb0bcb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:DLQ.queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b00c0000 nid=0x410e in Object.wait() [0x00007f71b524d000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb0b1e8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb0b1e8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://DLQ.queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b00be800 nid=0x410d in Object.wait() [0x00007f71b534e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb0a5e8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb0a5e8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b00bc000 nid=0x410c waiting on condition [0x00007f71b544f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efad75c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:example.A" daemon prio=10 tid=0x00007f71b04c5000 nid=0x410b in Object.wait() [0x00007f71b5550000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efad6588> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efad6588> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://example.A" daemon prio=10 tid=0x00007f71b04c3000 nid=0x410a in Object.wait() [0x00007f71b5651000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efad59d8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efad59d8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b04c8000 nid=0x4109 waiting on condition [0x00007f71b5752000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71efb09a10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:queue/Einarbeitungsdaten_RemoteQ" daemon prio=10 tid=0x00007f71b04c7000 nid=0x4108 in Object.wait() [0x00007f71b5853000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb08f40> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb08f40> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://queue/Einarbeitungsdaten_RemoteQ" daemon prio=10 tid=0x00007f71b04c6000 nid=0x4107 in Object.wait() [0x00007f71b5954000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb08330> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb08330> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Topic  ActiveMQ.Advisory.Queue" daemon prio=10 tid=0x00007f71b0613800 nid=0x4106 in Object.wait() [0x00007f71b5a55000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efb487a8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efb487a8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Producer Flow Control Timeout Task" prio=10 tid=0x00007f71b04c9800 nid=0x4105 waiting on condition [0x00007f71b5b56000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c3cbded0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask.run(Queue.java:169)

"Queue:queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b061e800 nid=0x4104 in Object.wait() [0x00007f71b5c57000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efac23b8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efac23b8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"Checkpoint: queue://queue/Initialdaten_RemoteQ" daemon prio=10 tid=0x00007f71b04b3800 nid=0x4103 in Object.wait() [0x00007f71b5d58000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efac1990> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71efac1990> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"ActiveMQ Journal Checkpoint Worker" daemon prio=10 tid=0x00007f71b05d0800 nid=0x4102 in Object.wait() [0x00007f71b6b72000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71ef960c48> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0x00007f71ef960c48> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

"RMI RenewClean-[10.130.18.244:35003]" daemon prio=10 tid=0x00007f71b04d8000 nid=0x4100 in Object.wait() [0x00007f71b5f5a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efc91e50> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00007f71efc91e50> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Data File Writer" daemon prio=10 tid=0x00007f71b83b6000 nid=0x40ff in Object.wait() [0x00007f71b655e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c7fd9fa0> (a org.apache.activemq.kaha.impl.async.DataFileAppender$1)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.kaha.impl.async.NIODataFileAppender.processQueue(NIODataFileAppender.java:80)
        - locked <0x00007f71c7fd9fa0> (a org.apache.activemq.kaha.impl.async.DataFileAppender$1)
        at org.apache.activemq.kaha.impl.async.DataFileAppender$2.run(DataFileAppender.java:230)

"RMI Scheduler(0)" daemon prio=10 tid=0x00007f71b83b7000 nid=0x40fe waiting on condition [0x00007f71b665f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f71c28fd860> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
        at java.util.concurrent.DelayQueue.take(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"GC Daemon" daemon prio=10 tid=0x0000000040564800 nid=0x40fc in Object.wait() [0x00007f71b6970000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2926ee0> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(Unknown Source)
        - locked <0x00007f71c2926ee0> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=10 tid=0x000000004023a000 nid=0x40fb in Object.wait() [0x00007f71b6a71000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c28fbee0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00007f71c28fbee0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at sun.rmi.transport.ObjectTable$Reaper.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Accept-51109" daemon prio=10 tid=0x00007f71b8333800 nid=0x40f9 runnable [0x00007f71b6c73000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c28817a8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"ActiveMQ Scheduler" daemon prio=10 tid=0x00007f71b836e800 nid=0x40f8 in Object.wait() [0x00007f71b6d74000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c28ba498> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x00007f71c28ba498> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f71b0092000 nid=0x40f6 runnable [0x00007f71b70c6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c2881d38> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Accept-11099" daemon prio=10 tid=0x00007f71b0090000 nid=0x40f5 runnable [0x00007f71b71c7000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c28821c0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f71b0082000 nid=0x40f4 runnable [0x00007f71b72c8000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x00007f71c2882630> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"CommunicatorServer" daemon prio=10 tid=0x00007f71b003c000 nid=0x40f3 runnable [0x00007f71b73c9000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked <0x00007f71c289e760> (a java.net.PlainDatagramSocketImpl)
        at java.net.PlainDatagramSocketImpl.receive(Unknown Source)
        - locked <0x00007f71c289e760> (a java.net.PlainDatagramSocketImpl)
        at java.net.DatagramSocket.receive(Unknown Source)
        - locked <0x00007f71c289e7d0> (a java.net.DatagramPacket)
        - locked <0x00007f71c289e718> (a java.net.DatagramSocket)
        at com.sun.jmx.snmp.daemon.SnmpAdaptorServer.doReceive(Unknown Source)
        at com.sun.jmx.snmp.daemon.CommunicatorServer.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Low Memory Detector" daemon prio=10 tid=0x00007f71b0002000 nid=0x40ef runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00007f71b826c000 nid=0x40ee waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00007f71b826a000 nid=0x40ed waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" daemon prio=10 tid=0x00007f71b8267800 nid=0x40ec runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x00007f71b8264000 nid=0x40eb runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f71b8258000 nid=0x40ea waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f71b823a000 nid=0x40e9 in Object.wait() [0x00007f71b7ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2927370> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00007f71c2927370> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x00007f71b8238000 nid=0x40e8 in Object.wait() [0x00007f71bc1af000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71c2926ed0> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x00007f71c2926ed0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x000000004011d800 nid=0x40d8 in Object.wait() [0x00007f71f6488000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71efd33780> (a [Z)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.activemq.console.command.StartCommand.waitForShutdown(StartCommand.java:161)
        - locked <0x00007f71efd33780> (a [Z)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:104)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
        at org.apache.activemq.console.Main.main(Main.java:107)

"VM Thread" prio=10 tid=0x00007f71b8233800 nid=0x40e7 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000040127800 nid=0x40da runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040129800 nid=0x40db runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000004012b800 nid=0x40dc runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x000000004012d000 nid=0x40dd runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x000000004012f000 nid=0x40de runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040131000 nid=0x40df runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000040132800 nid=0x40e0 runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000040134800 nid=0x40e1 runnable 

"GC task thread#8 (ParallelGC)" prio=10 tid=0x0000000040136800 nid=0x40e2 runnable 

"GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040138000 nid=0x40e3 runnable 

"GC task thread#10 (ParallelGC)" prio=10 tid=0x000000004013a000 nid=0x40e4 runnable 

"GC task thread#11 (ParallelGC)" prio=10 tid=0x000000004013c000 nid=0x40e5 runnable 

"GC task thread#12 (ParallelGC)" prio=10 tid=0x000000004013d800 nid=0x40e6 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007f71b0094800 nid=0x40f7 waiting on condition 

JNI global references: 6239

Heap
 PSYoungGen      total 199552K, used 51797K [0x00007f71e2800000, 0x00007f71f1f10000, 0x00007f71f2800000)
  eden space 158528K, 23% used [0x00007f71e2800000,0x00007f71e4bdcd18,0x00007f71ec2d0000)
  from space 41024K, 36% used [0x00007f71ef700000,0x00007f71f05b8820,0x00007f71f1f10000)
  to   space 47232K, 0% used [0x00007f71ec2d0000,0x00007f71ec2d0000,0x00007f71ef0f0000)
 PSOldGen        total 199616K, used 95454K [0x00007f71c2800000, 0x00007f71ceaf0000, 0x00007f71e2800000)
  object space 199616K, 47% used [0x00007f71c2800000,0x00007f71c8537b48,0x00007f71ceaf0000)
 PSPermGen       total 43264K, used 43152K [0x00007f71bd400000, 0x00007f71bfe40000, 0x00007f71c2800000)
  object space 43264K, 99% used [0x00007f71bd400000,0x00007f71bfe24138,0x00007f71bfe40000)

 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent flooding queue://queue/Einarbeitungsdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 210s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-12587-1270572565654-0:22380:1:1) stopped to prevent flooding queue://queue/Initialdaten_RemoteQ. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 300s)
 INFO | Usage Manager Store is Full. Producer (ID:HOST2-46261-1270635495961-0:0:6:1) stopped to prevent floodi
{code} 

  
> Producer Flow Control seems to be broken
> ----------------------------------------
>
>                 Key: AMQ-2680
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2680
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker, Documentation
>    Affects Versions: 5.3.1
>         Environment: SLES 11, IBM JDK 1.6
>            Reporter: Marc Schöchlin
>            Assignee: Gary Tully
>            Priority: Critical
>             Fix For: 5.3.1
>
>         Attachments: activemq-broken-flow-control.xml, activemq.xml
>
>
> How to reproduce:
> - Download ActiveMQ 5.3.1
> - Start broker with the attached configuration
>    ---
>    <systemUsage>
>        <systemUsage>
>            <memoryUsage>
>                <memoryUsage limit="10 mb"/>
>            </memoryUsage>
>            <storeUsage>
>                <storeUsage limit="10 mb" name="queue/RemoteQ"/>
>            </storeUsage>
>            <tempUsage>
>                <tempUsage limit="3 gb"/>
>            </tempUsage>
>        </systemUsage>
>    </systemUsage>
>    ---
> - Start example producer/consumer
>    cd examples
>    ant producer -Durl=tcp://127.0.0.1:61616 -Dmax=100000 -Dsubject=queue/RemoteQ -Ddurable=true -DsleepTime=2
>    sleep 600
>    ant consumer -Durl=tcp://127.0.0.1:61616  -Dsubject=queue/RemoteQ -Dmax=100000 -DsleepTime=2
> The problem: 
> When the 10mb limit is reached, the producer is blocked forever.
> The expected behavior is, that if a consumer cleans the queue, producers are able to post more messages.
> Neither cleaning up the queue, restarting producers and restarting the broker helps :-(
> It would be also a good idea to enhance the documentation about producer-flow-control:
> (http://activemq.apache.org/producer-flow-control.html)
> - expected behavior
> - what is limited in detail by using system|store|tempUsage 
> The entire configuration :
> ---
> <beans
>   xmlns="http://www.springframework.org/schema/beans"
>   xmlns:amq="http://activemq.apache.org/schema/core"
>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>   xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>   http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
>     <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>             <value>file:${activemq.base}/conf/credentials.properties</value>
>         </property>      
>     </bean>
>     <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}/data" >
>         <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <policyEntry topic=">" producerFlowControl="true" memoryLimit="10mb">
>                   <deadLetterStrategy>
>                     <!--
>                       Use the prefix 'DLQ.' for the destination name, and make
>                       the DLQ a queue rather than a topic
>                     -->
>                     <individualDeadLetterStrategy
>                       queuePrefix="DLQ." useQueueForQueueMessages="true" />
>                   </deadLetterStrategy>
>                   <pendingSubscriberPolicy>
>                     <vmCursor />
>                   </pendingSubscriberPolicy>
>                 </policyEntry>
>                 <policyEntry queue=">" producerFlowControl="true" memoryLimit="10mb">
>                   <!-- Use VM cursor for better latency
>                        For more information, see:
>                        
>                        http://activemq.apache.org/message-cursors.html
>                        
>                   <pendingQueuePolicy>
>                     <vmQueueCursor/>
>                   </pendingQueuePolicy>
>                   -->
>                   <deadLetterStrategy>
>                     <!--
>                       Use the prefix 'DLQ.' for the destination name, and make
>                       the DLQ a queue rather than a topic
>                     -->
>                     <individualDeadLetterStrategy
>                       queuePrefix="DLQ." useQueueForQueueMessages="true" />
>                   </deadLetterStrategy>
>                 </policyEntry>
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy> 
>  
>         <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage limit="10 mb"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="10 mb" name="queue/RemoteQ"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="3 gb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>                   
>         <transportConnectors>
>             <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
>         </transportConnectors>
>     </broker>
>     <import resource="jetty.xml"/>
>     
> </beans>
> ---

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message