activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: JMX connections creating high cpu and GC
Date Wed, 11 Nov 2015 06:41:02 GMT
Do you see the same performance impact from attaching JConsole or JVisualVM?
On Nov 10, 2015 4:09 PM, "Basmajian, Raffi" <rbasmajian@ofiglobal.com>
wrote:

> I'm throwing a hail mary on this one,
>
> We've set up a broker cluster on A-MQ 5.11 (Fuse 6.2).
> Six master/slave pairs, full graph topology network of brokers; 12 brokers
> total.
> The cluster is brand new, no message activity; network connectors are
> active and working properly.
> Java 8, RHEL 7.1, 1gb/4Gb min/max
>
> Problem
> =======
> Using JMC (Mission Control) , we connect to a master to view JMX metrics.
> Almost immediately, cpu activity on the broker shoots to 100%. Heap
> consumption oscillates between 128Mb and 800Mb over 30s windows, and the
> rest of the day is generally unpleasant.
>
> From a thread view, nearly all cpu activity is consumed by numerous
> "ActiveMQ Task-xxx" threads; here's the stack trace from one, but they are
> all nearly identical:
>
> ActiveMQ Task-220 [2656] (TIMED_WAITING)
>    sun.misc.Unsafe.park line: not available [native method]
>    java.util.concurrent.locks.LockSupport.parkNanos line: 215
>    java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill line:
> 460
>    java.util.concurrent.SynchronousQueue$TransferStack.transfer line: 362
>    java.util.concurrent.SynchronousQueue.poll line: 941
>    java.util.concurrent.ThreadPoolExecutor.getTask line: 1066
>    java.util.concurrent.ThreadPoolExecutor.runWorker line: 1127
>    java.util.concurrent.ThreadPoolExecutor$Worker.run line: 617
>    java.lang.Thread.run line: 745
>
> At first I thought it was related to this issue, and while I found five
> threads with the name "JMX Server connection timeout" on this instance,
> none of those threads are consuming cpu resources.
> https://access.redhat.com/solutions/1169753
>
> When we test this on a standalone instance with no network configuration,
> this problem does not occur. In our QA environment (the config detailed
> above), the broker config is identical to our standalone instances, the
> only difference is the network connector. Here's a snippet, there are five
> total pairs like this (for six total m/s pairs in the topology) in each
> activemq.xml:
>
>             <!--        Network #1        -->
>             <!-- (#1) Queue network link  -->
>             <networkConnector
>                 name="queues_nc1"
>                 userName="${auth.user}"
>                 password="${auth.password}"
>                 uri="masterslave(tcp://whatever, tcp://whatever)"
>                 consumerTTL="1"
>                 messageTTL="100"
>                 conduitSubscriptions="false"
>                 decreaseNetworkConsumerPriority="true"
>                 suppressDuplicateQueueSubscriptions="true">
>                 <dynamicallyIncludedDestinations>
>                     <queue physicalName=">"/>
>                 </dynamicallyIncludedDestinations>
>             </networkConnector>
>             <!-- (#1) Topic network link  -->
>             <networkConnector
>                 name="topics_nc1 "
>                 userName="${auth.user}"
>                 password="${auth.password}"
>                 uri="masterslave(tcp://whatever, tcp://whatever)"
>                 consumerTTL="1"
>                 messageTTL="100 "
>                 decreaseNetworkConsumerPriority="true">
>                 <dynamicallyIncludedDestinations>
>                     <topic physicalName=">"/>
>                 </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>
>
> This e-mail transmission may contain information that is proprietary,
> privileged and/or confidential and is intended exclusively for the
> person(s) to whom it is addressed. Any use, copying, retention or
> disclosure by any person other than the intended recipient or the intended
> recipient's designees is strictly prohibited. If you are not the intended
> recipient or their designee, please notify the sender immediately by return
> e-mail and delete all copies. OppenheimerFunds may, at its sole discretion,
> monitor, review, retain and/or disclose the content of all email
> communications.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message