activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: ActiveMQ 5.5.0 broker crashing
Date Thu, 14 Jul 2011 10:30:21 GMT
That looks like a memory leak. There is a potential issue with
inactive destination deletion that has been resolved on trunk that can
be worked around by setting schedulePeriodForDestinationPurge=0 on the
broker element in xml configuration.

Otherwise we need to track down the component that is using up your
heap. I guess there are two approaches. Try and reduce your use or
activemq to a minimum such that you can reproduce the OOM  and raise a
jira issue with the test case or use some tools like yourkit or jmap
on the broker to figure out the culprit.

On the latter, there are some nice resource on the web like:
http://olex.openlogic.com/wazi/2009/how-to-fix-memory-leaks-in-java/

Once the JMV gets its first OOM, there will be lots of errors in the
logs, activemq is not recoverable in that state without a restart.

On 14 July 2011 10:49, Will <will@sonnex.name> wrote:
> I have a problem whereby the ActiveMQ broker crashes when my system has been
> running for more than ~6 hours. By "crashes" I mean it's still running but
> won't allow any sending/receiving of messages and needs to be restarted.
> Here's my setup:
>
> ActiveMQ 5.5.0 with the out-of-the-box configuration, except after it
> crashed the first few times I set Xmx to 2048M and
> UseDedicatedTaskRunner=false, not really sure why though.
>
> Has around 200 (non-transacted) connections from locally networked PCs but
> very minimal traffic - around two messages per second.
>
> Runs on Ubuntu 9.04, using:
> java version "1.6.0_16"
> Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>
>
> My data/activemq.log shows the following when it crashes:
>
> 2011-07-14 03:57:54,605 | INFO  | Transport failed: java.io.EOFException |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transpo
> rt: tcp:///146.169.52.132:53248
> 2011-07-14 03:58:00,925 | INFO  | Transport failed: java.io.EOFException |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transpo
> rt: tcp:///146.169.52.133:50171
> 2011-07-14 03:58:03,795 | INFO  | Transport failed:
> java.net.SocketException: Broken pipe |
> org.apache.activemq.broker.TransportConnection.Transport |
>  InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@4dcd6d55
> 2011-07-14 03:58:07,275 | INFO  | Transport failed:
> java.net.SocketException: Broken pipe |
> org.apache.activemq.broker.TransportConnection.Transport |
>  Async Exception Handler
> 2011-07-14 03:58:07,274 | INFO  | Transport failed:
> java.net.SocketException: Broken pipe |
> org.apache.activemq.broker.TransportConnection.Transport |
>  InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@5ff907ee
> 2011-07-14 03:58:03,802 | INFO  | Transport failed:
> java.net.SocketException: Broken pipe |
> org.apache.activemq.broker.TransportConnection.Transport |
>  InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@3bd1c463
> 2011-07-14 03:58:07,274 | INFO  | Transport failed:
> java.net.SocketException: Broken pipe |
> org.apache.activemq.broker.TransportConnection.Transport |
>  InactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@3a15e82a
>
>
> etc. etc. lots more Broken pipes... then some of these interspersed with
> many more broken pipes:
>
> 2011-07-14 03:58:10,201 | INFO  | Transport failed:
> org.apache.activemq.transport.InactivityIOException: Cannot send, channel
> has already failed: /146
> .169.53.87:46663 | org.apache.activemq.broker.TransportConnection.Transport
> | Async Exception Handler
>
>
> Then some of these (still interspersed with broken pipes and failed channel
> messages):
>
> 2011-07-14 04:18:50,633 | WARN  | Failed to unregister mbean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType
> =address,Name=/146.169.52.126_33912 |
> org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ
> Task-103
>
>
> Then the end is nigh with a few of these:
>
> 2011-07-14 04:19:54,077 | WARN  | Failed to remove consumer: ID:"I removed
> the address here"-60248-1310591381109-0:1:-1:1. Reason:
> java.lang.OutOfMemoryError
> : Java heap space | org.apache.activemq.broker.TransportConnection |
> ActiveMQ Task-69
> java.lang.OutOfMemoryError: Java heap space
>        at
> java.util.concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:481)
>        at
> java.util.concurrent.CopyOnWriteArraySet.remove(CopyOnWriteArraySet.java:205)
>        at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.unregisterSubscription(ManagedRegionBroker.java:381)
>        at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.removeConsumer(ManagedRegionBroker.java:255)
>        at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
>        at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:225)
>        at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
>        at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
>        at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:123)
>        at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:575)
>        at
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:613)
>        at
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:726)
>        at
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1025)
>        at
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:71)
>        at
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:952)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
> 2011-07-14 04:19:54,077 | WARN  | Problem scavenging sessions |
> org.eclipse.jetty.util.log | HashSessionScavenger-1
> java.lang.OutOfMemoryError: Java heap space
>
>
> After the broker "crashes" my clients dump the following when attempting to
> create a temporary queue (though "createTemporaryQueue" is just the first
> method of the local "Session" that they call):
>
> Exception in thread "Thread-4" javax.jms.JMSException: Timer already
> cancelled.
>        at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>        at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1295)
>        at
> org.apache.activemq.ActiveMQConnection.createTempDestination(ActiveMQConnection.java:1930)
>        at
> org.apache.activemq.ActiveMQSession.createTemporaryQueue(ActiveMQSession.java:1330)
>        at project.hub.Client.receive(Client.scala:54)
>        at project.Node$.colony(Node.scala:52)
>        at
> project.Node$$anonfun$main$1$$anonfun$apply$1.apply$mcV$sp(Node.scala:69)
>        at project.util.Ops$$anon$1.run(Ops.scala:5)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.IllegalStateException: Timer already cancelled.
>        at java.util.Timer.sched(Timer.java:354)
>        at java.util.Timer.schedule(Timer.java:222)
>        at
> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>        at org.apache.activemq.broker.region.Queue.start(Queue.java:852)
>        at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:129)
>        at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:304)
>        at
> org.apache.activemq.broker.region.RegionBroker.addDestinationInfo(RegionBroker.java:367)
>        at
> org.apache.activemq.advisory.AdvisoryBroker.addDestinationInfo(AdvisoryBroker.java:160)
>        at
> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:217)
>        at
> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:217)
>        at
> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
>        at
> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:480)
>        at
> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
>        at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
>        at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>        at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>        at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>        at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
>        at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>        at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
>        at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
>        at java.lang.Thread.run(Thread.java:619)
>
>
>
> I also get the following from a different client thread that is blocked on
> "MessageConsumer.receive":
>
> Exception in thread "Thread-10" javax.jms.JMSException: Channel was inactive
> for too (>30000) long: server/146.169.2.83:61678
>        at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
>        at
> org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:458)
>        at
> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
>        at project.hub.Client$$anonfun$2.apply$mcV$sp(Client.scala:32)
>        at project.util.Ops$$anon$1.run(Ops.scala:5)
>        at java.lang.Thread.run(Thread.java:636)
>
>
> Another type of broker client I have dumps these:
>
> Exception in thread "Thread-10" javax.jms.JMSException: Cannot send, channel
> has already failed: server/146.169.2.83:61678
>        at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
>        at
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1267)
>        at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
>        at
> org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1863)
>        at
> org.apache.activemq.ActiveMQMessageProducer.close(ActiveMQMessageProducer.java:147)
>        at project.hub.Server$.theoryProvider(Server.scala:114)
>        at
> project.hub.Server$$anonfun$main$2$$anonfun$apply$1.apply$mcV$sp(Server.scala:176)
>        at project.util.Ops$$anon$1.run(Ops.scala:5)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
> channel has already failed: server/146.169.2.83:61678
>        at
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:246)
>        at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
>        at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
>        at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>        at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>        at
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
>        ... 7 more
>
>
>
> Any help would be greatly appreciated.
>
> Will
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-0-broker-crashing-tp3667090p3667090.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
http://fusesource.com
http://blog.garytully.com

Mime
View raw message