activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Hoover <roger.hoo...@gmail.com>
Subject Re: Gradually increasing, huge cpu-load when using Stomp+nio
Date Wed, 20 Jan 2010 22:06:36 GMT
Hi Arjen,

We had a similar setup with lots of PHP processes as producers making STOMP
connections, enqueuing a message, and disconnecting and experienced many
problems including max file descriptor errors and other broker issues.
 We've seen much better stability and throughput by introducing a proxy so
that the PHP producers do an HTTP POST to the proxy instead of connecting
directly AMQ with STOMP.  The proxy caches the message temporarily to local
filesystem and a daemon with a single, long-lived STOMP connection picks up
the files and enqueues them.

We use nginx + fastcgi for the proxy. I see this as a nice separation of
concerns.  IMO, AMQ is not optimized for handling lots of short-lived
connections but nginx is (or any other async multiplexer) and can easily
handle thousands of concurrent connections per process.  So, using this
setup allows nginx to do what it does well (very efficiently multiplex
connections) and AMQ to do what it does well (pass messages between
producers and consumers with long-lived connections).

HTH,

Roger

On Wed, Jan 20, 2010 at 1:08 PM, Arjen van der Meijden <
acmmailing@tweakers.net> wrote:

> Hi List,
>
> I just restarted a activemq 5.3-instance which was causing 1000% cpu (the
> machine has 8 cores, so all where fully working) and a load of 1035.
>
> I'm pretty confident that this load/cpu-usage was caused by using stomp+nio
> rather than 'stomp' (whether it was just nio or not, I don't know).
>
> Our usage-pattern is that we have a few queues with a single consumer each.
> The producers are php-web-processes that make a connection using stomp, send
> a few messages to some of the queues and disconnect again (which is why we
> have the transport.closeAsync=false). The heaviest two queues receive about
> 30-50 messages/second during peak hours (and thus, ActiveMQ receives about
> 30-50 connections/second).
>
> We also have a few topics that get messages quite infrequently, from
> similar stomp-producers, but those are consumed with java-based
> applications.
>
> The load/cpu-usage apparently has gradually increased, as can be seen in
> these two graphs:
> Cpu-usage
>
> http://tweakers.net/stats/?Action=Generator&Mode=Serverstats&Time=1263919627&Dagen=1&StatsServer=Argus&colServers=CPUUsage
>
> Load average
>
> http://tweakers.net/stats/?Action=Generator&Mode=Serverstats&Time=1263919627&Dagen=1&StatsServer=Argus&colServers=LoadAvg
>
> The increase in load has gone steadily in just a few hours up untill the
> moment where it took the php-processes more than 13 seconds to actually
> connect, send two messages and disconnect (rather than a few milliseconds).
>
> Our activemq.xml is attached, although I already replaced stomp+nio with
> normal stomp. Our ACTIVEM_OPTS is "-Xms2048M -Xmx2048M -XX:+UseParallelOldGC
> -server -Dorg.apache.activemq.UseDedicatedTaskRunner=true" (i.e. the default
> with adjusted memory and GC)
>
> The increasing load hasn't appeared yet with the adjusted config that uses
> non-nio connections.
>
> Unfortunately, this is a production server so I can't run tests very
> easily. Nor do I have a problem with not running nio, so this is mainly a
> mail to let you know of this problem, rather than me having a big issue with
> activemq right now :)
>
> Best regards,
>
> Arjen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message