The master/slave configuration for AMQ should address your HA concerns. In your case, I think a shared file system or jdbc master/slave would be the way to go. Have you used the maven 2 or jmeter performance tests for benchmarking? http://activemq.apache.org/jmeter-performance-tests.html http://activemq.apache.org/activemq-performance-module-users-manual.html With the proper settings (e.g., async sends, prefetch limits, etc.) you'll hopefully be pleasantly surprised as to what one AMQ broker can deliver d:) Best of luck and hope AMQ works out for you. Joe Marc Zampetti wrote: > > Joe, > > Thanks for the suggestion. But in this case, it won't work since the > routing criteria are much too fine-grained. Basically, each of the 6 > million messages will each have a unique id. Then some subset, say 1 > million messages, will need to be routed to a process for special > processing. Basically, what I have is a list of those 1 million ids that I > need to look for. I'm thinking I will need to write a simple consumer that > acts as the router, reading in the routing list and consuming the > messages, determining if they match, and then publishing them on the final > destination. My concern is scalability and HA in that situation. I think > using an embedded broker might help, doing something similar to what Camel > is trying to do. > > One of my concerns here is the likely hood that I might overwhelm a > broker. I only have a small number of publishers, so I'm concerned with > too many publishers ending up on the same broker. I'm also concerned with > the shear number of messages that would have to be forwarded to the other > brokers in the network in this situation. > > Marc > > > ttmdev wrote: >> >> Re the second part of your post. If Camel is not an option, then what >> about a composite queue in combination with selectors? For example, in >> the snippet below, Q.FOO gets a subset of the message stream being sent >> to Q.BLAST, while Q.BAR gets the entire stream. >> >> >> >> > /> >> >> >> >> >> Hope this helps - Joe >> >> >> Marc Zampetti wrote: >>> >>> All, >>> >>> I'm considering ActiveMQ for an application that has very high message >>> rates expected, at the rate of 6 - 10 million messages per minute. All >>> of these messages are fairly small, on the order of 100 bytes or less, >>> but they will be very regular, with a a large burst of additional >>> messages (around 20 million extra) once an hour. Obviously, I'm looking >>> at a fairly large Network of Brokers. I don't expect, nor do I need >>> persistent messages on disk, nor do I want guaranteed delivery, though >>> it would be nice. :-) Does anyone have any idea if this is even possible >>> with AMQ? >>> >>> There are a few portions of the applications that need to receive a >>> subset of the message stream, and other portions that will simply >>> process the entire stream. For those components that need to get a >>> sub-set, I need to have some way to route the appropriate messages to >>> the components. While still only a subset, this could still be 1 >>> million+ messages per minute, and I'm looking for an efficient way to >>> decide when to route a message or not. Each of these 6 million messages >>> are unique, with a unique identifier, so I would need to have an id to >>> queue mapping table in order to perform the routing. At 1 million+, my >>> concern is that the table itself can get pretty large, and that some of >>> the more "normal" routing things that Camel might help with won't be >>> that helpful. >>> >>> Anyone have any ideas or best practices? >>> >>> Marc >>> >> >> > > -- View this message in context: http://www.nabble.com/Questions-on-Network-of-Brokers-and-high-message-rates-tf4941283s2354.html#a14172452 Sent from the ActiveMQ - User mailing list archive at Nabble.com.