activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Washusen <...@reactive.org>
Subject performance issues with slow consumers
Date Wed, 14 Nov 2007 06:50:58 GMT

Hey All,
I'm currently load testing our application with ActiveMQ and I'm seeing some
strange behavior that I'm hoping someone can offer some assistance with.  

Basically, there are 7 machines participating in the load test;
1. Running JMeter.
2. Running JMeter.
3. Running ActiveMQ 4.1.1. (mel1u105).
4. Running Producer (mel1u108).
5. Running Producer (mel1u109).
6. Running Consumer (mel1u110).
7. Running Consumer (mel1u111).
8. Running PostgreSQL (mel1u113).

ActiveMQ is allocated 1500Mb of memory running on a AMD64 X2, 3.5GB RAM,
~350GB Raid 5 type machine.  For a period the producers are adding around
1000 messages per second to a queue until (according to the JMX console)
there are around 600000 messages queued.  At this point ActiveMQ basically
locks up and consumers slow down to a trickle.  

I've attached CPU utilization graphs of each of the nodes and as you can see
the node running ActiveMQ keeps consuming CPU time while all the other nodes
basically idle (the other nodes do more than just interact with ActiveMQ).

I've tried this with both journaledJDBC and kahaPersistenceAdapter (see
attached activemq.xml) to basically the same result.  I was hoping that the
persistence adapter would handle the peak periods when the producers are
creating messages at a much higher rate than they can be consumed.  

So my questions are;
1. Is there a limit to the number of messages ActiveMQ can queue even with
disk based persistence? 
2. Can someone suggest the configuration that would handle the above?  It's
very important that the producers be allowed to operate as normal (public
facing) and if necessary we could have ActiveMQ ditch the messages it can't
deal with.

Also, please note that when the consumers can keep up with the producers
everything works fine.

http://www.nabble.com/file/p13740826/sar_cpu_mel1u105.sar.sadf.png 
http://www.nabble.com/file/p13740826/sar_cpu_mel1u108.sar.sadf.png 
http://www.nabble.com/file/p13740826/sar_cpu_mel1u109.sar.sadf.png 
http://www.nabble.com/file/p13740826/sar_cpu_mel1u110.sar.sadf.png 
http://www.nabble.com/file/p13740826/sar_cpu_mel1u111.sar.sadf.png 
http://www.nabble.com/file/p13740826/sar_cpu_mel1u113.sar.sadf.png 

http://www.nabble.com/file/p13740826/activemq.xml activemq.xml 
-- 
View this message in context: http://www.nabble.com/performance-issues-with-slow-consumers-tf4802510s2354.html#a13740826
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message