activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will <w...@sonnex.name>
Subject ActiveMQ 5.5.0 broker crashing
Date Thu, 14 Jul 2011 09:49:03 GMT
I have a problem whereby the ActiveMQ broker crashes when my system has been
running for more than ~6 hours. By "crashes" I mean it's still running but
won't allow any sending/receiving of messages and needs to be restarted.
Here's my setup:

ActiveMQ 5.5.0 with the out-of-the-box configuration, except after it
crashed the first few times I set Xmx to 2048M and
UseDedicatedTaskRunner=false, not really sure why though.

Has around 200 (non-transacted) connections from locally networked PCs but
very minimal traffic - around two messages per second.

Runs on Ubuntu 9.04, using:
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)


My data/activemq.log shows the following when it crashes:

2011-07-14 03:57:54,605 | INFO  | Transport failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transpo
rt: tcp:///146.169.52.132:53248
2011-07-14 03:58:00,925 | INFO  | Transport failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transpo
rt: tcp:///146.169.52.133:50171
2011-07-14 03:58:03,795 | INFO  | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
 InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@4dcd6d55
2011-07-14 03:58:07,275 | INFO  | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
 Async Exception Handler
2011-07-14 03:58:07,274 | INFO  | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
 InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@5ff907ee
2011-07-14 03:58:03,802 | INFO  | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
 InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@3bd1c463
2011-07-14 03:58:07,274 | INFO  | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
 InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@3a15e82a


etc. etc. lots more Broken pipes... then some of these interspersed with
many more broken pipes:

2011-07-14 03:58:10,201 | INFO  | Transport failed:
org.apache.activemq.transport.InactivityIOException: Cannot send, channel
has already failed: /146
.169.53.87:46663 | org.apache.activemq.broker.TransportConnection.Transport
| Async Exception Handler


Then some of these (still interspersed with broken pipes and failed channel
messages):

2011-07-14 04:18:50,633 | WARN  | Failed to unregister mbean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType
=address,Name=/146.169.52.126_33912 |
org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ
Task-103


Then the end is nigh with a few of these:

2011-07-14 04:19:54,077 | WARN  | Failed to remove consumer: ID:"I removed
the address here"-60248-1310591381109-0:1:-1:1. Reason:
java.lang.OutOfMemoryError
: Java heap space | org.apache.activemq.broker.TransportConnection |
ActiveMQ Task-69
java.lang.OutOfMemoryError: Java heap space
        at
java.util.concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:481)
        at
java.util.concurrent.CopyOnWriteArraySet.remove(CopyOnWriteArraySet.java:205)
        at
org.apache.activemq.broker.jmx.ManagedRegionBroker.unregisterSubscription(ManagedRegionBroker.java:381)
        at
org.apache.activemq.broker.jmx.ManagedRegionBroker.removeConsumer(ManagedRegionBroker.java:255)
        at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
        at
org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:225)
        at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
        at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
        at
org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:123)
        at
org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:575)
        at
org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:613)
        at
org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:726)
        at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1025)
        at
org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:71)
        at
org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:952)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
2011-07-14 04:19:54,077 | WARN  | Problem scavenging sessions |
org.eclipse.jetty.util.log | HashSessionScavenger-1
java.lang.OutOfMemoryError: Java heap space


After the broker "crashes" my clients dump the following when attempting to
create a temporary queue (though "createTemporaryQueue" is just the first
method of the local "Session" that they call):

Exception in thread "Thread-4" javax.jms.JMSException: Timer already
cancelled.
        at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
        at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1295)
        at
org.apache.activemq.ActiveMQConnection.createTempDestination(ActiveMQConnection.java:1930)
        at
org.apache.activemq.ActiveMQSession.createTemporaryQueue(ActiveMQSession.java:1330)
        at project.hub.Client.receive(Client.scala:54)
        at project.Node$.colony(Node.scala:52)
        at
project.Node$$anonfun$main$1$$anonfun$apply$1.apply$mcV$sp(Node.scala:69)
        at project.util.Ops$$anon$1.run(Ops.scala:5)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.IllegalStateException: Timer already cancelled.
        at java.util.Timer.sched(Timer.java:354)
        at java.util.Timer.schedule(Timer.java:222)
        at
org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
        at org.apache.activemq.broker.region.Queue.start(Queue.java:852)
        at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:129)
        at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:304)
        at
org.apache.activemq.broker.region.RegionBroker.addDestinationInfo(RegionBroker.java:367)
        at
org.apache.activemq.advisory.AdvisoryBroker.addDestinationInfo(AdvisoryBroker.java:160)
        at
org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:217)
        at
org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:217)
        at
org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
        at
org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:480)
        at
org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
        at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
        at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
        at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
        at java.lang.Thread.run(Thread.java:619)



I also get the following from a different client thread that is blocked on
"MessageConsumer.receive":

Exception in thread "Thread-10" javax.jms.JMSException: Channel was inactive
for too (>30000) long: server/146.169.2.83:61678
        at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
        at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:458)
        at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
        at project.hub.Client$$anonfun$2.apply$mcV$sp(Client.scala:32)
        at project.util.Ops$$anon$1.run(Ops.scala:5)
        at java.lang.Thread.run(Thread.java:636)


Another type of broker client I have dumps these:

Exception in thread "Thread-10" javax.jms.JMSException: Cannot send, channel
has already failed: server/146.169.2.83:61678
        at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
        at
org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1267)
        at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
        at
org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1863)
        at
org.apache.activemq.ActiveMQMessageProducer.close(ActiveMQMessageProducer.java:147)
        at project.hub.Server$.theoryProvider(Server.scala:114)
        at
project.hub.Server$$anonfun$main$2$$anonfun$apply$1.apply$mcV$sp(Server.scala:176)
        at project.util.Ops$$anon$1.run(Ops.scala:5)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
channel has already failed: server/146.169.2.83:61678
        at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:246)
        at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
        at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
        at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at
org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
        ... 7 more



Any help would be greatly appreciated.

Will

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-0-broker-crashing-tp3667090p3667090.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message