directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <>
Subject Re: Replication & activeMQ
Date Wed, 10 Aug 2011 16:16:08 GMT
On Wed, Aug 10, 2011 at 9:12 PM, Emmanuel Lecharny <> wrote:
> Hi guys,
> currently, we are using ActiveMQ to store the modifications we send to the
> client. This leads to an issue caused by the way we have configured it,
> simply because all the mods are stored in memory, and never removed.
> Obviously, this is bad.
> Thinking about it, my opinion is that it's may be a bit overkilling
yes, indeed, please see further below for the rationale behind this decision
> considering our need :
> - when a mod is made on the provider, it has to be sent to the consumer
> - in any case, we store the mod in a file associated with the consumer
> - we send the mod to the consumer unless we *know* that the consumer is
> offline
> - we will have no way to be sure that the consumer has correctly received
> the mod
> - when a consumer gets online again, it will send a cookie with the last CSN
> it received
> - in this case, we have to get all the mods from the file, and send the mods
> to the consumer.
and this is where the things turn out to be complex (definitely not impossible)
> Now, we already are saving the mods in a file, in the JournalInterceptor. We
> just have to implement the recovery system (ie, finding the last entry sent
> from the file) and send all the following entries. Then we can delete all
> the entries older than the requested one.
> I don't think that using our own implementation would be an issue here. I
> mean, ActiveMQ is a great piece of software, but having to go through tends
> of options (hundreds?), most of which are totally useless in our case, is a
> bit overkilling for this version.
> thoughts ?
the syncrepl protocol provides a high level of granularity about the
kind of data that can be
replicated, so not each mod in this journal is necessarily replicated
to a client (it can even be
a serious issue to send to that client based on the sensitivity of data).

This brings the issue of filtering the data that needs to be sent to
the client, this requires significant
time for scanning, processing and maintaining the position pointers in
the monolithic journal.

ActiveMQ's core offers all these features so that was a preferred
choice instead of writing a journal with all the above
mentioned features, cause implementing such a journal is quite a
handful of work and our main problem to be solved
is replication.

Having said that ActiveMQ is definitely an over kill to be used as a
journal, the main part that we actually need is
it's journal/store implementation called 'kahadb' but it is easy to
use through high level message queue interfaces.

I would like to spend some time on 'kahadb' source to see if that can
be easily embeddable and serves our purpose.

By no means am against a new implementation but I prefer to use some
battle tested component (for which kahadb perfectly
qualifies) if available.

Kiran Ayyagari

View raw message