activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Zampetti <zampet...@aim.com>
Subject Re: Questions on Network of Brokers and high message rates
Date Wed, 05 Dec 2007 16:03:01 GMT

James,

Yes, it sounds like the JEDI thing and the partitioning approach is what I
need. And yes, I'm talking queues for the most part.  For the partitioning,
is that something that AMQ, or are you talking about me having a layer in
front of AMQ that would do this. In this case, instead of having one big
network of brokers, I would have several smaller networks of brokers?

Marc


James.Strachan wrote:
> 
> On 05/12/2007, Marc Zampetti <zampettim@aim.com> wrote:
>> James,
>>
>> You are right about the HA comment I originally made. I was referring to
>> fact that I'm not looking for persistent messages. But I am concerned
>> about
>> what happens when a broker fails and being able to recover from that
>> quickly, even if that means losing messages.
> 
> Master/Slave is all about replicating messages to avoid message loss -
> it comes at a heavy performance cost. If you can't afford that cost,
> you don't need master/slave. i.e. the normal failover of the JMS
> client will do fine.
> 
> 
>> I understand the point about the need for more information. Here is what
>> I
>> say so far.
>>
>> There will be around 30 - 50 processes producing messages, each one
>> producing the same type of messages. Think it of a farm of processes
>> doing
>> the same basic work, spread out to handle processing load, scalability,
>> fault tolerance, etc. These processes know nothing about what happens to
>> these messages once they send them.
>>
>> On the consuming side, there are multiple farms of processes, each farm
>> doing a particular type of processing on some subset of the messages.
>> There
>> is nothing in the message, other than a unique id, that can be used to
>> determine which messages should be sent to each farm. Each of these farms
>> would have anywhere from 2 - 30 processes, depending upon the
>> functionality
>> and load requirements.
> 
> So we're talking queues right? And by the sounds of it you could
> partition messages across multiple brokers for scalability. (Sounds
> like a good use case for the JEDI transport that still needs
> completing BTW to stripe message flows across different brokers to
> avoid the downsides of the store and forward network of brokers).
> 
> 
>> For the AMQ topologies, I was thinking of a Virtual Topic that all of the
>> publishers send messages to, with queues behind that go to some sort of
>> routing layer that in turns sends the message to the queue that the
>> service
>> the relevant consumer farm.
> 
> Yeah. I guess it depends on how complex the routing is. It might be
> easier to just have an input queue, then some consumer process (which
> could be hosted inside the broker if need be) to do the routing;
> whether its Camel or some custom JMS code or whatever.
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Questions-on-Network-of-Brokers-and-high-message-rates-tf4941283s2354.html#a14174653
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message