activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Heitmann <jheitm...@gmail.com>
Subject Mitigating uneven queue consumption
Date Mon, 19 Jun 2006 18:30:27 GMT
Our setup: We have a set of brokers directly feeding consumers  
messages on a single queue destination (I'm going to lie a bit about  
our setup to keep things simple). On each host we're running a broker  
and multiple consumers. Each consumer connects to the broker set with  
a failover connection. Messages flow into the broker set from  
upstream sources (in our case another set of brokers) in a pretty  
even fashion, so each broker gets about the same amount of messages.

The problem with this is that it's easy for one broker to have an  
uneven number of consumers, and in some cases have so few consumers  
that it will backlog. One way this can happen is if a broker is taken  
down for some routine reason. All the connections to that broker will  
fail over to another broker. When the broker comes back up it will  
not receive new connections until consumers restart yet it will  
continue to receive an equal share of messages from upstream.

We're considering implementing two different code changes to ActiveMQ  
to help with this, but I wanted to poll the group to be sure there  
was not already something there to help us.

Our first solution would be to add logic to ActiveMQConnection that  
would trigger a reconnect on the failover transport after a certain  
configurable time period expires and the current transaction is  
complete. This would give decent statistical certainty that a single  
broker won't ever get a large backlog compared to the others and  
should be transparent to the client code except for a hitch in  
message flow.

Our second solution is to implement connection multiplexing for  
consumers. Behind the scenes each consumer would connect to all of  
the broker set, divide its prefetch buffer across all those brokers,  
and start grabbing messages from multiple at once. We'd also need  
hooks to auto-discovery for when we bring new brokers up or old  
brokers come back. Also transactions would probably need to be tied  
to a single broker. This seems like it would be smoother behavior  
overall, but tougher to implement than the first so we're putting it  
off. Scalability is also a concern.

If you know of a way to do this already I would appreciate hearing  
about it. Thanks,

John

Mime
View raw message