qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Queue mirroring: clustering vs. replication
Date Fri, 19 Jul 2013 15:15:39 GMT
On 07/17/2013 10:17 AM, Steve Huston wrote:
> Hi Steve,
>
>> -----Original Message-----
>> From: Rothkin, Steve (NY81) [mailto:Steve.Rothkin@Honeywell.com]
>> Sent: Wednesday, July 17, 2013 8:57 AM
>> To: users@qpid.apache.org
>> Subject: Queue mirroring: clustering vs. replication
>>
>>>> Subject: RE: Queue mirroring with message grouping and clustering on
>>>> Windows
>>>>
>>>>>> I'm considering a Qpid.22 implementation under MS Windows for
>>>>>> message queueing.  In the future we might go to a mixed
>>>>>> environment with both Windows and Linux computers.
>>>>>> For fault tolerance, I want the queues to be mirrored across 2
>>>>>> to
>>>>>> 3 computers which are connected by high speed LAN. Each queue
>>>>>> will have multiple consumers on different computers (including
>>>>>> some NOT hosting the queue), so I also need to use the message
>>>>>> grouping feature to ensure that messages from a single source
>>>>>> are not processed
>>>> out of order.
>>
>> What are the differences between choosing to use clustering to implement
>> the solution vs. using ha-replication of individual queues in a non-clustered
>> setup?
>
> You summarized it well below, but also, essentially, reusing the capabilities already
present, or reinventing them to suit the particulars of your use case.
>
>>  From what I've read so far it seems that with clustering my work would be a
>> lot easier as the broker/resource manager combination would take care of
>> failover while with non-clustered I'd have to take care of a bunch of things in
>> my code -- designating master, detecting failure (which might take longer),
>> failover (including finding and connecting to the new master).
>
> Right. Don't forget to account for all the ways things can fail. Brokers, nodes, network
links, etc.
>
>> On the other hand it seems that without clustering, I could pick different
>> masters for different queues (instead of having cluster resource manager
>> designate a specific host as master for all of its queues) which would be one
>> way of spreading the load of handling client connections.
>
> You could also run multiple clustered brokers on the same nodes and have broker-cluster-1
serve one set of queues and broker-cluster-2 serve a different set. I'm not sure about the
automatability of keeping the primaries on different nodes, but you could write a script to
check it and move the primary around if needed.
>
>> I could also have
>> different operating systems hosting replicas of the same queue (I haven't
>> found a cluster resource manager that runs on both Windows and Linux).
>
> That's a great point - clusters are generally uniform.
>
>> I'm also wondering about geographic site mirroring and failover. We have 2
>> sites connected by a WAN. The same services are available at both locations.
>> Normally a request received at one location would be processed there (and
>> sending it to the other site for processing would be more "expensive" as
>> would queue mirroring to the other site). But there might be situations
>> where one entire site goes down -- if that were to happen it might be good
>> to have the other site detect and finish processing any in-flight messages
>> from the down site.
>
> There are more factors in this type of design than I'd want to try and cover in email.
I'm working with a customer now that is starting to run tests on a multi-site environment,
but their goals and constraints and resources may be quite different from yours. I think you've
got a good grasp of the facilities available (queue dup, federation, clustering) but if you
want further in-depth help, please let me know.
>
> -Steve

Steve's done a great job with the clustering facts, I don't have much to add. If 
you get interested in integrating another resource manager let me know, I'd like 
to help. I need to do an integration with pacemaker in the near future. Also any 
other issues you have, you can discuss them on this list or raise a JIRA if you 
have a specific bug that needs fixing.

Cheers,
Alan.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message