activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Van Dewoestine <ervan...@gmail.com>
Subject Issue creating a distributed queue using store and forward.
Date Wed, 07 Oct 2009 17:09:58 GMT
ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)

I'm running into an issue with the store and forward feature of
activemq, which I using in an attempt to create a highly available
distributed queue.  I'm trying to figure out if the issue is a
misconfiguration on my part, expected behavior of activemq, or a bug.

The summary of the problem is that given 2 brokers, B1 and B2, which
each have one consumer, C1 and C2, which are subscribed to the same
queue.  If I stop a consumer on one of the brokers, the pending
messages from that broker are not always forwarded to the other broker
which still has a consumer, leading to those messages getting
indefinitely stuck.

The steps I use to reproduce this scenario are as follows (Note: all
producing and consuming is performed over the stomp transport):

Couple notes about the consumers:
- they have a prefetchSize of 40
- the processing of messages can take some time, so for the purposes
  of this exercise, I've created a simple consumer that sleeps for 10
  seconds before sending the message ack (using client-individual ack
  mode)

1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
   yet running.
2. produce a few thousand messages to B1
   Note: B1 now has a few thousand pending messages and B2 has 0.
3. start consumer C2 (listing for messages from B2)
   Note: messages are are successfully received and begin processing
   (monitoring the brokers shows pending messages decreasing). Now B2
   has all the pending messages and B1 has 0.
4. start consumer C1 (listing for messages from B1)
   Note: no messages are received, which is another issue I have since
   B2 now has thousands of pending messages which C1 could help
   process, but instead sits idle while C2 is forced to handle all the
   messages.
5. stop consumer C2
   Note: now I have thousands of messages sitting on B2 and 0 on B1
   where a C1 is alive and ready to handle them.  So at this point,
   despite having a consumer running, thousands of messages are stuck
   in the queue.
6. stop consumer C1
   Note: now I have no consumers.  Stopping and restarting C1 has no
   effect on the pending messages sitting on B1's queue.
7. stop both brokers
8. start B1, then start B2
9. start C1
   Note: now all messages have migrated from B2 to B1 and C1 is again
   processing messages.

So after step 5, the only way to recover from the stuck messages is to
restart the brokers.

Below is my current connector config which I have on both brokers.
I've tried playing with the various properties of the connector, but
it seems as though no matter what I try the above scenario continues
to occur.

<networkConnector
    name="default-nc"
    uri="multicast://default?group=${broker.group}"
    dynamicOnly="true"
    networkTTL="25"
    suppressDuplicateQueueSubscriptions="true"/>


So, is this an activemq bug? Am I mis-using activemq? Is there some
other way to achieve a highly available distributed queue?

Any help in this regard is greatly appreciated.

--
eric

Mime
View raw message