activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MassDosage <massdos...@gmail.com>
Subject Re: Options for preferring stability over reliability
Date Thu, 02 Apr 2009 22:57:45 GMT

Thanks for the quick response. We are running 2 instances of ActiveMQ - one
is running 5.1.0 and the other is running 5.2.0. The one running 5.1.0 runs
out of memory every now and then and all connected senders go nuts and we
see a *big* increase in CPU on the sender machines (possibly due to high
number of blocked senders?) I will try to a jstack on both the sender
process and the ActiveMQ process next time this happens.

The 5.2.0 setup (see attached activemq.xml) is the one that stops responding
completely with nothing immediately suspicious in the logs. The last time
this happened I did a jstack on our sending application and there were
*many* entries like this:

"pool-1-thread-38" prio=10 tid=0x0000000000aa7800 nid=0x7c6b waiting for
monitor entry [0x0000000044745000..0x0000000044745aa0]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1599)
        - waiting to lock <0x00002b0f421e1e10> (a java.lang.Object)
        at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
        at
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
        at fm.last.jms.Publisher.publish(Publisher.java:82)
.....

The sending machine's CPU and memory were maxed out, probably due to the JVM
having to handle all these thousands of blocked threads. We are using
sendFailIfNoSpace but I had no idea that was mutually exclusive with turning
flow control off. We would like to avoid flow control if we can as this has
caused us problems in the past. 

If there is anything else I can provide you with please let me know. I will
try get jstack output for the ActiveMQ processes the next time we have
issues. I'm afraid this isn't something that we can't reproduce in a unit
test as it really seems to happen unpredictably when the system has been
running for a while. It's not even that it always happens during peak load
times. http://www.nabble.com/file/p22858692/activemq.xml activemq.xml 
-- 
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p22858692.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message