activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: ActiveMQ broker OOM
Date Thu, 19 Oct 2017 13:25:15 GMT
1) All the overhead data structures of the broker, including the running
threads. 2) The memory store, which is used for both non-persistent
messages and for paging persistent messages back in from the persistent
store. The paging is done per-queue if I remember correctly, so if you have
lots of persistent queues with only a few messages each, you could be
trying to page in most of the messages from the persistent store.

You could try reducing the percentOfJvmHeap parameter to force the memory
store to overflow to the temp store sooner (leaving more space for the
overhead data structures), or you could give the broker more heap. Or if
you think something might be wrong (something could be, though it's
impossible to tell from the information you've given so far and no one else
has reported such a problem), you could use a memory profiler to try to
determine what memory is being allocated so we can see if there's a memory
leak of some sort.

BTW, what GC algorithm are you using? If it's CMS, try switching to G1,
since CMS has a possible failure mode due to memory fragmentation that G1
isn't subject to.

Tim

On Sun, Oct 1, 2017 at 7:13 AM, eladko <elad@skyfence.com> wrote:

> Hi all,
>
> I'm getting OOM error in my broker:
>
> INFO   | jvm 1    | 2017/09/30 13:39:42 |  INFO | Queue.34324.LIST Inactive
> for longer than 60000 ms - removing ...
> INFO   | jvm 1    | 2017/09/30 15:34:04 |  WARN | Store COMMIT FAILED:
> INFO   | jvm 1    | 2017/09/30 15:34:04 | java.lang.OutOfMemoryError: GC
> overhead limit exceeded
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.util.DataByteArrayOutputStream.<
> init>(DataByteArrayOutputStream.java:47)[activemq-kahadb-
> store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.page.Transaction$
> 1.<init>(Transaction.java:279)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.page.Transaction.
> openOutputStream(Transaction.java:279)[activemq-kahadb-
> store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.page.Transaction.
> store(Transaction.java:256)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.
> storeNode(BTreeIndex.java:293)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode.
> put(BTreeNode.java:394)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode.
> put(BTreeNode.java:377)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.
> put(BTreeIndex.java:189)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase$MessageOrderIndex.put(
> MessageDatabase.java:3692)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase.
> updateIndex(MessageDatabase.java:1462)[activemq-kahadb-
> store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase$AddOperation.execute(
> MessageDatabase.java:3106)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase$16.
> execute(MessageDatabase.java:1391)[activemq-kahadb-store-5.
> 15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.disk.page.Transaction.
> execute(Transaction.java:779)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase.
> process(MessageDatabase.java:1387)[activemq-kahadb-store-5.
> 15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase$10.
> visit(MessageDatabase.java:1226)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.data.KahaCommitCommand.
> visit(KahaCommitCommand.java:130)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase.
> process(MessageDatabase.java:1208)[activemq-kahadb-store-5.
> 15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase.store(
> MessageDatabase.java:1123)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.MessageDatabase.store(
> MessageDatabase.java:1099)[activemq-kahadb-store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore.
> commit(KahaDBTransactionStore.java:301)[activemq-kahadb-
> store-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.transaction.LocalTransaction.
> commit(LocalTransaction.java:70)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.broker.TransactionBroker.commitTransaction(
> TransactionBroker.java:252)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.broker.BrokerFilter.commitTransaction(
> BrokerFilter.java:114)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.broker.BrokerFilter.commitTransaction(
> BrokerFilter.java:114)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.broker.TransportConnection.
> processCommitTransactionOnePhase(TransportConnection.java:
> 529)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.command.TransactionInfo.visit(
> TransactionInfo.java:100)[activemq-client-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.broker.TransportConnection.service(
> TransportConnection.java:330)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.broker.TransportConnection$1.
> onCommand(TransportConnection.java:194)[activemq-broker-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.transport.MutexTransport.onCommand(
> MutexTransport.java:50)[activemq-client-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(
> WireFormatNegotiator.java:125)[activemq-client-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(
> AbstractInactivityMonitor.java:301)[activemq-client-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:04 |       at
> org.apache.activemq.transport.TransportSupport.doConsume(
> TransportSupport.java:83)[activemq-client-5.15.0.jar:5.15.0]
> INFO   | jvm 1    | 2017/09/30 15:34:13 | Exception in thread "ActiveMQ
> Connection Dispatcher: tcp://11.1.1.77:43506" java.lang.OutOfMemoryError:
> GC
> overhead limit exceeded
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.openwire.BooleanStream.<init>(BooleanStream.java:26)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.openwire.OpenWireFormat.marshal(
> OpenWireFormat.java:225)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(
> TcpTransport.java:193)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(
> AbstractInactivityMonitor.java:335)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(
> AbstractInactivityMonitor.java:317)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.transport.TransportFilter.oneway(
> TransportFilter.java:94)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(
> WireFormatNegotiator.java:116)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.transport.MutexTransport.oneway(
> MutexTransport.java:68)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.broker.TransportConnection.dispatch(
> TransportConnection.java:1486)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.broker.TransportConnection.processDispatch(
> TransportConnection.java:971)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.broker.TransportConnection.iterate(
> TransportConnection.java:1021)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(
> DedicatedTaskRunner.java:112)
> INFO   | jvm 1    | 2017/09/30 15:34:13 |       at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(
> DedicatedTaskRunner.java:42)
> INFO   | jvm 1    | 2017/09/30 15:34:19 |  WARN | Async error occurred:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO   | jvm 1    | 2017/09/30 15:34:28 |  WARN | Async error occurred:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO   | jvm 1    | 2017/09/30 15:39:42 |  INFO | Queue.13757.LIST Inactive
> for longer than 60000 ms - removing ...
>
> I'm using persisted queues, and I can understand why the heap is getting so
> big.
> <persistenceAdapter>
>             <kahaDB directory="${activemq.data}/kahadb"
> enableJournalDiskSyncs="false"/>
> </persistenceAdapter>
>
> Both Xmx and Xms are 2048MB, and my config in activemq.xml is:
> <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage percentOfJvmHeap="70" />
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="246 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="50 gb"/>
>                 </tempUsage>
>             </systemUsage>
> </systemUsage>
>
> Can someone explain what's being stored in the memory that can cause this
> OOM?
> Thanks all.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message