activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ttmdev <joe.fernan...@ttmsolutions.com>
Subject Re: Questions on Network of Brokers and high message rates
Date Wed, 05 Dec 2007 14:17:01 GMT

The master/slave configuration for AMQ should address your HA concerns. In
your case, I think a shared file system or jdbc master/slave would be the
way to go. 

Have you used the maven 2 or jmeter performance tests for benchmarking?  

http://activemq.apache.org/jmeter-performance-tests.html
http://activemq.apache.org/activemq-performance-module-users-manual.html

With the proper settings (e.g., async sends, prefetch limits, etc.) you'll
hopefully be pleasantly surprised as to what one AMQ broker can deliver d:) 

Best of luck and hope AMQ works out for you. 

Joe




Marc Zampetti wrote:
> 
> Joe,
> 
> Thanks for the suggestion. But in this case, it won't work since the
> routing criteria are much too fine-grained. Basically, each of the 6
> million messages will each have a unique id. Then some subset, say 1
> million messages, will need to be routed to a process for special
> processing. Basically, what I have is a list of those 1 million ids that I
> need to look for. I'm thinking I will need to write a simple consumer that
> acts as the router, reading in the routing list and consuming the
> messages, determining if they match, and then publishing them on the final
> destination. My concern is scalability and HA in that situation. I think
> using an embedded broker might help, doing something similar to what Camel
> is trying to do. 
> 
> One of my concerns here is the likely hood that I might overwhelm a
> broker. I only have a small number of publishers, so I'm concerned with
> too many publishers ending up on the same broker. I'm also concerned with
> the shear number of messages that would have to be forwarded to the other
> brokers in the network in this situation.
> 
> Marc 
> 
> 
> ttmdev wrote:
>> 
>> Re the second part of your post. If Camel is not an option, then what
>> about a composite queue in combination with selectors?  For example, in
>> the snippet below, Q.FOO gets a subset of the message stream being sent
>> to Q.BLAST, while Q.BAR gets the entire stream.  
>> 
>> <compositeQueue name="Q.BLAST">
>>             <forwardTo>
>>               <filteredDestination selector=”color=’blue’” queue="Q.FOO"
>> />
>>               <queue physicalName="Q.BAR" />              
>>             </forwardTo>
>> </compositeQueue>
>> 
>> Hope this helps - Joe
>> 
>> 
>> Marc Zampetti wrote:
>>> 
>>> All,
>>> 
>>> I'm considering ActiveMQ for an application that has very high message
>>> rates expected, at the rate of 6 - 10 million messages per minute. All
>>> of these messages are fairly small, on the order of 100 bytes or less,
>>> but they will be very regular, with a a large burst of additional
>>> messages (around 20 million extra) once an hour. Obviously, I'm looking
>>> at a fairly large Network of Brokers. I don't expect, nor do I need
>>> persistent messages on disk, nor do I want guaranteed delivery, though
>>> it would be nice. :-) Does anyone have any idea if this is even possible
>>> with AMQ? 
>>> 
>>> There are a few portions of the applications that need to receive a
>>> subset of the message stream, and other portions that will simply
>>> process the entire stream. For those components that need to get a
>>> sub-set, I need to have some way to route the appropriate messages to
>>> the components. While still only a subset, this could still be 1
>>> million+ messages per minute, and I'm looking for an efficient way to
>>> decide when to route a message or not. Each of these 6 million messages
>>> are unique, with a unique identifier, so I would need to have an id to
>>> queue mapping table in order to perform the routing. At 1 million+, my
>>> concern is that the table itself can get pretty large, and that some of
>>> the more "normal" routing things that Camel might help with won't be
>>> that helpful.
>>> 
>>> Anyone have any ideas or best practices?
>>> 
>>> Marc 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Questions-on-Network-of-Brokers-and-high-message-rates-tf4941283s2354.html#a14172452
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message