qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wright <atwri...@mac.com>
Subject Re: Java atomic bindings Was: Re: Queue creation/deletion
Date Mon, 02 Feb 2009 19:26:49 GMT
Thanks for the comments - a few responses below.

On 31 Jan 2009, at 23:20, Robert Greig wrote:

> 2009/1/30 Andrew Wright <atwright@mac.com>:
>
>> Actually this was to integrate with Terracotta. TC needs some sort  
>> of load
>> balancer to get locality of reference (for performance) - qpid  
>> would fit the
>> bill here, via hierarchical topics. There'd only be a single  
>> subscriber on a
>> topic at a time. TC clients are notified as nodes join/leave, so  
>> can take
>> over the processing of a given topic if the primary processor goes  
>> away.
>
> So you have a number of nodes which access Terracotta shared storage
> and you want to push jobs into those nodes, with some kind of affinity
> between job types and nodes in order to be able to get the locality of
> reference.
>

Essentially, yes. Each app node is responsible for a subset of the  
data processing (we're looking to implement an order matching engine/ 
exchange).

> The issue with using a topic is that recovery is made more complicated
> - unless the backup also subscribes to the messages and just throws
> them away. I would be tempted to use a queue with the headers exchange
> and just load balance between the primary and secondary (effective
> having active/active pairs with the clients). I'm obviously making a
> lot of assumptions about your application's suitability for that
> approach!
>

The big difficulty we have is a hard requirement for in-order message  
processing. This unfortunately rules out load balancing from a queue  
across active-active client pairs. We'd rather just route incoming  
msgs to a store-and-forward topic, and accept a small processing pause  
during app node failover.

Once a message has been read from the topic, it is written into the TC  
cache. At that point we can failover processing to a different node  
without any message or data loss.


> As far as the broker goes, I think that is an orthogonal question. I
> would run active/active brokers behind a hardware load balancer such
> as an F5. That will mean the client doesn't have to deal with failover
> plus you don't have to use shared storage so should give higher
> throughput (Carl can you confirm?).
>

We were rather hoping that qpid clients (eg. the Java client) would be  
able to failover against a HA broker pair without the use of a  
dedicated load balancer.

>> As you say, Coherence has this sort of thing built in.  
>> Unfortunately it also
>> comes with a fairly hefty Oracle price tag...
>
> Yes it is expensive but it is a much more sophisticated product (or
> certainly a "different" product) than Terracotta - I think it would
> depend what your non-functionals are whether it is worth using it (or
> Gemfire, IBM ObjectGrid or Gigaspaces which are credible Coherence
> competitors).

Absolutely - and we haven't ruled anything out yet. We're going to  
start with the lowest cost option, and if it doesn't work out, then  
have a rethink.

Cheers,
Andrew

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message