qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Etienne Antoniutti Di Muro <etienne.antoniu...@adaptivebytes.com>
Subject Re: Implementing clustering support for Qpid Java broker
Date Wed, 15 Sep 2010 21:56:34 GMT
Hello Lahiru,

I am willing to start a discussion on how to implement java broker replication,
However I do agree with Rob, tho, we should spend some time first to
address some basic questions.

Here's my  initial feeling.

Goal:
What do we want to achieve with replication ? I'd suggest both  HA &
performance, that with some adaptation policies I do not consider nor
mutually exclusive, nor requiring too much compromise.

Approach:
Active/passive ? I'd suggest, as long as the state of the replicate
object(s) is clearly defined, go with passive replication of a
serialized instance of the state. It can always be recreated at
failover.

How:
Well, primary-backup by means of virtual synchrony via group communication.

Architecture:
Define a replication layer of middleware that provides replication
services (obviously), plugged into the broker architecture, well,
where it is more convenient. I do not know anything about Qpid
architecture, is there any document anywhere ?

However I do have some proposals, you can have a look here
http://portal.acm.org/citation.cfm?id=1101143

As a starter, I'd need someone to help me understand Qpid
architecture, at least an overview.

Regards
Etienne


On Wed, Sep 15, 2010 at 11:20 AM, Lahiru Gunathilake <glahiru@gmail.com> wrote:
> Hi Etienne,
>
> I would like to start doing some work with Active/Passive clustering as a
> initial clustering support for Java broker since according to Rob
> implementing Active/Active clustering is bit tricky due to a design issues
> in Java broker. For my requirements with our products we can live with
> active/passive clustering since we can configure our Enterprise Service bus
> to use the passive node to be used only in a failover conditions so we
> really do not need to use active/active clustering to address the HA issue
> when we are using Apache Qpid with our solutions.
>
> I would like to start a discussion on how to implement Active/Passive
> clustering implementation for Java broker.
>
> Rob can you please initiate a discussion on this if you think this will be a
> useful thing for Java broker.
>
> Lahiru
>
> On Tue, Sep 14, 2010 at 4:52 AM, Etienne Antoniutti <
> etienne.antoniutti@adaptivebytes.com> wrote:
>
>> Hello Everybody,
>>
>> sorry, noticed this just now, while cleaning my inbox.
>>
>> Is there any further development on this topic ?
>> I'd might be interested in being involved on java clustering.
>>
>> Regards
>> Etienne
>>
>> >----- Original Message -----
>> >From: glahiru@gmail.com
>> >To: dev@qpid.apache.org
>> >Data: Wed, 7 Jul 2010 14:59:43 +0530
>> >Subject: Re: Implementing clustering support for Qpid Java broker
>>
>> >Hi Rob,
>> >
>> >Yes true I agree that we need to look in to the architecture and look on
>> how
>> >should we implement clustering and I just post about the library since
>> Sorin
>> >just asked about the library and I am currently looking in to the broker
>> >architecture before deciding anything and I hope to discuss every step in
>> >the list before implementing anything.
>> >
>> >I am thinking of listing what are we going to achieve by clustering Java
>> >broker. Then we can discuss on where should we start and what is the first
>> >version of clustering support should do. Then we can work on it and make
>> >sure all of them are successfully implemented.
>> >
>> >Currently what I am thinking is achieving HA without loosing messages
>> which
>> >is the basic requirement of any clustering approach.
>> >
>> >Thanks a lot Robert for your reply. can you please elaborate little bit on
>> >why should we wait until the AMQP 1-0 support is finished, if there's a
>> >pretty good reason for that we can wait until it comes up.
>> >
>> >Lahiru
>> >
>> >On Wed, Jul 7, 2010 at 2:47 PM, Robert Godfrey <rob.j.godfrey@gmail.com
>> >wrote:
>> >
>> >> First of all we need to be clear about what we mean / want to achieve
>> >> by "clustering".
>> >>
>> >> If we are talking about clustering for reliability/HA then we need to
>> >> be very clear that the limitations AMQP 0-8/0-9/0-9-1 prevent us from
>> >> actually implementing this unless we mandate that every message
>> >> published into the broker is inside a transaction (due to the fact
>> >> that published messages are not acknowledges for AMQP 0-x versions
>> >> where x < 10).
>> >>
>> >> The next issue is that the Java broker has not really been designed
>> >> with clustering in mind - it's therefore not clear inside the code
>> >> which pieces of state need to be replicated / where the broker should
>> >> wait to hear that state has been successfully replicated before
>> >> proceeding.
>> >>
>> >> My personal view on clustering is that we need to spend a fair amount
>> >> of time thinking about the internal architecture of the broker before
>> >> we start worrying about what other libraries / tools we pull in.
>> >>
>> >> There are probably ways of doing and active-passive cluster relatively
>> >> cheaply (essentially replicating the store to the passive broker) -
>> >> I'm not sure if this would help your use case Lahiru.
>> >>
>> >> Having said all the above, I am very keen to add clustering
>> >> functionality to the Java Broker, and I've some vague notion that I
>> >> should start looking at this later in the year.  Personally I'd be
>> >> looking at doing it after whatever re-architecting is done for
>> >> supporting AMQP 1-0.
>> >>
>> >> Sorry if I sound discouraging - and if someone can come along and show
>> >> an easy way to add clustering, I'd be impressed.
>> >>
>> >> Cheers,
>> >> Rob
>> >>
>> >> On 7 July 2010 00:30, Lahiru Gunathilake <glahiru@gmail.com> wrote:
>> >> > Hi devs,
>> >> >
>> >> > Last couple of weeks I was working with Qpid Java broker in an actual
>> >> > production environment and I encounter a scenario where we should
>> cluster
>> >> > Qpid and since we do not have a native clustering support we found
a
>> >> > solution to the single point of failure of Qpid but it is not an
>> actual
>> >> > cluster since we do not replicate the messages among nodes. So I would
>> be
>> >> > really happy to work in a clustering implementation for Qpid.
>> >> >
>> >> > Appreciate your ideas on this and I will update the list once we
>> discuss
>> >> a
>> >> > proper design for clustering implementation.
>> >> >
>> >> > Regards
>> >> > Lahiru
>> >> >
>> >> > --
>> >> > Apache Qpid, Worlds dominant messaging middleware..!!!
>> >> >
>> >> > Senior Software Engineer
>> >> > WSO2 Inc
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> Apache Qpid - AMQP Messaging Implementation
>> >> Project:      http://qpid.apache.org
>> >> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>> >>
>> >>
>> >
>> >
>> >--
>> >Apache Qpid, Worlds dominant messaging middleware..!!!
>> >
>> >Senior Software Engineer
>> >WSO2 Inc
>> >
>>
>>
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


Mime
View raw message