directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@gmail.com>
Subject Re: [Mitosis] Push or pul, plus some random thoughts
Date Thu, 15 Jan 2009 00:28:31 GMT
Howard Chu wrote:
> Emmanuel Lecharny wrote:
>> Hi guys,
>>
>> the current implementation of replication, AFAICT, is based on a pull
>> system. So to speak, each time you modify some entry, a message is sent
>> to all the replicas, waiting for an ack to be returned.
>>
>> The main advantage is that we can't be faster: we replicate as soon as
>> possible.
>>
>> The main issue is that if a replica is not connected, we will try and
>> try and try, until the remote server is back.
>
> That can become a serious PITA; it certainly was with OpenLDAP slurpd.
It is already a PITA for Mitosis ;)

>
>> Here are some ideas I rehashed in the train on my way to office those
>> last two days...
>>
>> - We should ask the replicas to register to the other servers using a
>> LDAP extended request
>> - then the server will push the modifications in a blocking queue for
>> each replicas
>> - the blocking queue is read by a thread and the modifications are
>> stored in a base, and sent to the replicas using a LDAP request, with a
>> control (Replication)
>> - the replica receives the modifications as simple LDAP requests, plus a
>> control, and deal with those, and send back a Ldap response with a
>> status, allowing the modification to be removed from the store.
>> - if the replica is disconnected (for any reason), the server does not
>> send anymore modifications to the replica, until the replica connects 
>> again.
>> - in this case, we simply restart the thread and send all the pending
>> modifications found in the store and in the queue to the replica.
>
> Likewise, a queue of pending mods can become a serious PITA if the 
> replica is offline for a long time.
The queue will be limited in size, obviously. I can even be empty, all 
the modification being stored on disk.

However, we don't necessarily have to keep a track of pending modify for 
disconnect replicas, as it's easy to know which entries has not been 
replicated since the last time the replica was on. The connecting 
replica could send the last entryCSN received, and then a search can be 
done on the server for every CSN with a higher CSN. Then you don't need 
to keep any modification on disk, as they are already stored in the DiT.


>
> See section 17.2.4.1 "Replacing Slurpd" here
> http://www.openldap.org/doc/admin24/replication.html
Damn, _know_, I see that my previous sentence is exactly what OpenLDAP 
does... Either I'm a smart guy, or -more probably - I already read this 
page and it remains somewhere in my fuzzy memory :)

I have to get some sleep, and start reading more instead of reinventing 
the wheel, I guess :).

Thanks Howard !

-- 
--
cordialement, regards,
Emmanuel L├ęcharny
www.iktek.com
directory.apache.org



Mime
View raw message