activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Slow consumer problem
Date Tue, 06 Jun 2006 06:47:15 GMT
On 6/6/06, D Howe <david.howe@sensis.com.au> wrote:
>
> We are getting similar problems in ActiveMQ 3.2.1.  We have a similar
> scenario to Frank, where our consumers may be down (e.g. for maintenance)
> but our producers will continue to produce messages.

Unfortunately 3.x is cannot handle large queues that don't fit in
memory - its only strategy once the RAM buffer is used up is to block
producers until the consumers catch up. So if you want to deal with
fast producers and slow consumers on durable queues well I'd recommend
upgrading to 4.0.


> I was expecting that with the amount of journal capacity and the Oracle
> database that I would be able to get in the millions of entries into the
> queue when there was no consumers, and that they would simply "queue up".

The reason is the design trade off of performance; we've designed
ActiveMQ to be very fast (rather than doing a database query every
time a message is required to be dispatched etc). So 3.x keeps the
pending messages on a queue in RAM. As it happens its amazingly
complex efficiently dispatching messages from a massive queue
efficiently to an arbitrary number of consumers who maybe using
filters so it took us until 4.x to truly figure out how to do it well,
fast and to deal with arbitrarily massive queues. The dispatch model
of massive queues is one of the major reasons for the changes in 4.x


> What actually happens is that around 60,000 messages I start getting lots of
> messages in the log:
>
> 2006-06-02 16:21:22,369 [Thread-0       ] WARN
> ientQueueBoundedMessageManager - memory limit low - forced to remove expired
> messages: publish.request
>
> Around about 65,000 messages I start getting lots of:
>
> 2006-06-02 17:19:05,159 [ocalport=61616]] WARN  MemoryBoundedQueue
> - Queue is full, waiting for it to be dequeued.
>
> At this point, my producer seems to lock up and everything stops.

Yes - you've hit the limitation of 3.x - I"d recommend going to 4.x.
We've gone tests with millions of messages on a queue and things still
work fine.


> Is there any way to configure ActiveMQ (3.2.x) to reliably queue up a number
> of messages, only limited by disk/database capacity?

No - you need to move to 4.x.

>  The application that
> we are developing could produce average loads of 30 messages/second, so
> 65,000 messages is nothing.  Even if we up the maxMem, we will still have a
> limit - and the bigger the maxMem, the worse the consequences are if we
> reach it as we will lose more messages.
>
> Upgrading to 4 is not an option for us at the moment, as we are in final
> stages of testing and don't want to have to re-test with a version that is
> not final release.

4.0 is final?

-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message