activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bbuzzard <Billy.Buzz...@bnsflogistics.com>
Subject Receiving NetworkBroker seems to be stuck
Date Mon, 25 Sep 2017 15:29:15 GMT
I'm using ActiveMQ-5.5.1 with a centralized broker feeding approximately
twenty other brokers via NetworkBridges.  All of the brokers except for one
is working perfectly and have been for years.  We recently moved our Data
Warehouse (DW) to the cloud and that broker seems to hang up and stop
communicating four or five times a day.

I've used JMX to remotely monitor the centralized broker (HUB) and the DW
broker.  The HUB continues to move files to/from all of the brokers except
for the DW.  The HUB, via JMX, reports that the DW NetworkBridge is down,
but the DW broker says the NetworkBridge is up.

I turned on transport tracing for both the HUB and the DW brokers and I can
clearly see the KeepAlive messages going to the DW broker and the responses
coming back until the HUB reports the NetworkBridge to the DW is down.  My
JMX connection to the HUB continues to work and Heap and Nonheap usage seem
well within design limits, but the JMX connection to the DW returns a
timeout.  

I then tried logging into the DW (Linux box) and tried to run TOP.  If took
almost a minute for the letters T, O, P, to echo back which suggested to me
that the box was under heavy cpu load.  Just prior to the timeout, the DW
JMX connection showed that Heap and Nonheap were within design limits.  May
supervisor asked two very valid questions: "How do I know if the DW Broker
did or did not use up heap if I cannot see heap usage via JMX?" and "Could
GC be stuck?".  We also noticed that all ActiveMQ logging ceases while the
broker is hung.

The DW broker is supposed to run continuously.  The DW itself instantiates
several very large one shot processes every ten minutes and I suspect that
this is what is causing the DW broker and JMX to hang.

Does anyone have experience troubleshooting a problem like this?  What
should I do to prove that the problem is either the ActiveMQ broker or the
processed that the DW is instantiating?  If someone has seen this problem
and fixed it, how did you fix it?

The only way I found to fix the hung broker is to execute an activemq
restart that times out after thirty seconds and then does a kill on the pid.






--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Mime
View raw message