activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabe Westmaas" <gabe.westm...@rackspace.com>
Subject RE: 1->N queue model for high volume traffic?
Date Wed, 23 Sep 2009 10:53:23 GMT
I have an application with similary requirements, and we considered this route as well.  Whether
or not there are problems with this really depends on how messages come in.  For example,
for us, we have the potential for hundreds of thousands of queues based on algorithm you described,
and not only would this require hundreds of thousands of consumers, typically 99.999% of those
would be idle while the other 0.001% have multiple messages waiting in their queues.  This
would also be surmountable, except that which 0.001% of the queues are active changes from
minute to minute.  Therefore we needed a solution that would allow us to connect 10-20 consumers
and they would pull from whichever queue happened to be active at that moment.

However, if you have a small maximum number of queues, and you can easily connect to all those
queues at once with your consumers, the solution you described could be fine.  Of course,
selectors should also be fine in this situation, although in my testing of selectors I seemed
to have problems when the number of messages in the queue rose over 1500 or so.  Maybe that
has been fixed since 5.1.0, or my tests were wrong.

Gabe

-----Original Message-----
From: "Robert Nicholson" <robert.nicholson@gmail.com>
Sent: Wednesday, September 23, 2009 1:40am
To: users@activemq.apache.org
Subject: 1->N queue model for high volume traffic?

Is there ever an advantage to doing the following when order of  
processing matters?

There is a need to preserve order based on some characteristic of the  
message.

Therefore multiple consumers from the queue without the use of  
selectors is not an appropriate solution.

Instead, single consumer of the queue but this consumer will hash/ 
dispatch based on the characteristic
of the message to other queues and those queues have concurrent  
listeners one for each queue. (order of processing within the queue is  
preserved)

In my tests that seems to be quicker than single consumer only based  
on approaches where I'm required to use
the database to keep track of which messages I've already processed  
and then acknowledge the message
explicitly after processing. If there's any outage my database working  
table reflects what is left to do upon
restart of the process. The downside is that there's significant  
overhead in maintaining this table.

In the above example the idea is that when you use a transacted  
session if there's an interruption in processing
that causes a rollback of the session then redelivery of those  
messages that were not processed will occur. Therefore it's
no longer necessary to maintain this working table.

With this approach you commit the session in the same underlying  
thread that is responsible for delivery
to your listener.

....

Which DMLC (Spring) scenario allows you to process messages off the  
queue but still be able to
know that if there's an interruption that redelivery of all  
unprocessed messages is guaranteed to occur?
Obviously anything that acknowledges the message before processing  
start is not a viable solution.






Mime
View raw message