activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jouni Mäki-Panula (JIRA) <j...@apache.org>
Subject [jira] [Created] (AMQ-5993) Purging messages leads to GC overhead limit exceeded
Date Thu, 01 Oct 2015 06:33:05 GMT
Jouni Mäki-Panula created AMQ-5993:
--------------------------------------

             Summary: Purging messages leads to GC overhead limit exceeded
                 Key: AMQ-5993
                 URL: https://issues.apache.org/jira/browse/AMQ-5993
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.11.1
            Reporter: Jouni Mäki-Panula
            Priority: Minor


Purging a large number of messages from a queue in ActiveMQ 5.11.1 seems to lead to GC overhead
limit exceeded error state.

I've witnessed this behavior twice during the last month. In the first case, 3 314 226 messages
were purged. 12 minutes later, the broker basically halted and log started to fill with messages
like these:

{noformat}
| WARN  | Transport Connection to: tcp://172.20.50.174:56005 failed: java.io.IOException:
GC overhead limit exceeded | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
NIO Worker 27070
| ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: GC
overhead limit exceeded | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport
Server Thread Handler: stomp+nio://0.0.0.0:61613?maximumConnections=1400&wireFormat.maxFrameSize=104857600&transport.keepAlive=true&transport.soLinger=30
| WARN  | Async error occurred:  | org.apache.activemq.broker.TransportConnection.Service
| ActiveMQ BrokerService[localhost] Task-22496
java.lang.OutOfMemoryError: GC overhead limit exceeded
| WARN  | Exception occurred processing:
null: java.lang.OutOfMemoryError: GC overhead limit exceeded | org.apache.activemq.transport.stomp.ProtocolConverter
| ActiveMQ BrokerService[localhost] Task-22506
{noformat}

In the second case, 2 380 871 messages were purged. This time it took a bit longer for the
garbage collection to kick in, but it did, causing similar error state 6 hours and 14 minutes
later:

{noformat}
| WARN  | Async error occurred:  | org.apache.activemq.broker.TransportConnection.Service
| ActiveMQ NIO Worker 16937
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:142)[activemq-kahadb-store-5.11.1.jar:5.11.1]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:627)[activemq-kahadb-store-5.11.1.jar:5.11.1]
        at org.apache.activemq.store.kahadb.plist.PListStoreImpl.write(PListStoreImpl.java:403)[activemq-kahadb-store-5.11.1.jar:5.11.1]
        at org.apache.activemq.store.kahadb.plist.PListImpl.addLast(PListImpl.java:100)[activemq-kahadb-store-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.flushToDisk(FilePendingMessageCursor.java:440)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:231)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1796)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.Queue.orderedCursorAdd(Queue.java:878)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:854)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.Queue.send(Queue.java:733)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.util.LoggingBrokerPlugin.send(LoggingBrokerPlugin.java:275)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:541)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)[activemq-client-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:45)[activemq-client-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:87)[activemq-stomp-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:199)[activemq-stomp-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.stomp.ProtocolConverter.onStompSend(ProtocolConverter.java:332)[activemq-stomp-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:245)[activemq-stomp-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:75)[activemq-stomp-5.11.1.jar:5.11.1]
| WARN  | Exception occurred processing:
null: java.lang.OutOfMemoryError: GC overhead limit exceeded | org.apache.activemq.transport.stomp.ProtocolConverter
| ActiveMQ BrokerService[localhost] Task-17265
| WARN  | Transport Connection to: tcp://172.20.50.179:50888 failed: java.io.IOException:
GC overhead limit exceeded | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
NIO Worker 17016
| ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: GC
overhead limit exceeded | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport
Server Thread Handler: stomp+nio://0.0.0.0:61613?maximumConnections=1400&wireFormat.maxFrameSize=104857600&transport.keepAlive=true&transport.soLinger=30
{noformat}

The only way out seemed to be restarting the broker.

This ActiveMQ instance is running on a server with 4 GB system memory and 1 GB heap size for
ActiveMQ. Heap wasn't a problem in neither of the cases. In normal usage, there is around
2.5 GB free memory when counting OS caches, but only around 150 MB actual free memory. OS
is RHEL 6.6.

The most obvious solution may be to simply increase system memory. Another workaround would
be to dequeue messages from the queue with a consumer instead of purging them.

But still it seems odd that ActiveMQ is able to handle millions of messages if they are dequeued,
but not if they are purged. Should the purge operation be improved somehow? And is increasing
system memory the best option here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message