activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Snyder <bruce.sny...@gmail.com>
Subject Re: Configuring Distributed Queues in Store/Forward Network
Date Fri, 08 Oct 2010 02:00:21 GMT
On Thu, Oct 7, 2010 at 2:36 PM, jpeng <jpeng@xmatters.com> wrote:
>
> I am trying to setup distributed queue with a network of 3 brokers. 1
> producer, 1 fast consumer and 1 slow consumer. I cannot figure out how to
> configure ActiveMQ distribute the messages properly. It seems that messages
> are dispatched to each consumer regardless of how fast a consumer processes
> a message.
>
> Here are some details on the configuration:
>  -Message are non-persistent.
>  -Producer flow control is turned off.
>  -Prefetch is set to 1
>
> ====
>
> Test Case:
>  -The producer populates the queue with 100 messages.
>  -The fast consumer takes 50ms to process each message in the queue.
>  -The fast consumer takes 500ms to process each message in the queue.
>
> Expected Result:
>  -The fast consumer will process more messages than the slow consumer
>
> Actual Result:
>  -The slow consumer 49 messages and fast consumer 51 messages.
>  - The fast consumer completed process well ahead of the slow consumer but
> the slow consumer was dispatched half of the messages.
>
> =====
>
> Here is the test class and the logs from execution.
>   http://activemq.2283324.n4.nabble.com/file/n2967501/amq.log amq.log
>   http://activemq.2283324.n4.nabble.com/file/n2967501/SharedQueueTest.java
> SharedQueueTest.java
>
> =====
>
> Anyone have any ideas on how to configure the distributed queue so that
> messages can be dispatched to consumers based upon how fast they can
> process?

The results you are seeing are by design from a combination of the
prefetch limit and the round robin dispatch policy.

The prefetch limit determines how many messages can be sent to a
consumer to be processed. The default value for the queue prefetch
limit is 1000. Oftentimes this limit is fine for fast consumers that
never slow down, but it can easily flood a slow consumer. This is why
we recommend adjusting the prefetch limit based on the consumer
behavior.

The round robin dispatch policy uses a round robin algorithm to
distribute messages equally amongst all queue consumers. If you want
to distribute messages based on a given consumer's processing speed,
you would need to create a custom dispatch policy to do so. One
disadvantage to this idea is that it would require a lookup from each
consumer's JMX properties to determine its current processing
capabilities which would slow down the dispatch process. I suppose you
could perform the lookup periodically instead of on every loop in
order to minimize the performance hit a bit.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder

Mime
View raw message