directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jencks <>
Subject Re: [DRS] Changelog improvements
Date Mon, 26 Jan 2009 07:58:52 GMT

On Jan 25, 2009, at 11:40 PM, Kiran Ayyagari wrote:

> hi Emmanuel,
> Emmanuel Lecharny wrote:
>> Hi guys,
>> it's essential that we have a decent Changelog system in order to  
>> be able to build a reliable server, plus use it for replication  
>> eventually.
>> The currengt changelog system has many advantages :
>> - it basically works
>> - it's damn fast
>> - it can be extended easily
>> But it also has some drawbacks :
>> - it currently stores changes info in memory
>> - it stores forward and revert changes (we don't need revert usually)
>> - it stores changes in LdifEntry which uses Attributes instead of  
>> the new Entry API
>> We need to extend the ChangeLog API to get able to handle :
>> - on disk storage
>> - Entry instead of Attrinutes
> didn't understand this. Atm, the LdifEntry used in changelog holds a  
> ClientEntry
> and adds all the attributes to it. You mean to use a ServerEntry here?
>> - only forward operations
>> - fast lookup for an entry by its DN, or entryCSN/UUID attributes.
> We need to allow duplicates in the DN index, there will be collisions.
> How do we access CSN/UUID I think, this JournalInterceptor will be  
> before the
> ReplicationInterceptor then we don't have access to CSN. One way  
> would be to
> create these values in JournalInterceptor and pass on to  
> ReplicationInterceptor
> wdyt?
>> We may also add a new interceptor for that purpose, namely a  
>> JournalInterceptor.
>> Nothing complicated though. My idea is to use a simple file, being  
>> rotated when we put some tag, plus some JDBM index around to be  
>> able to lookup for DN, UUID or entryCSN.
> have started experimenting with a simple RandomAccessFile based  
> Journal
> ( but the work is quite slow cause of me being lazy ;) )

I'm not sure exactly what the requirements here are but a couple of  
existing projects that might provide a usable journal/log:

objectweb howl is a transaction log, basically its designed to write  
"prepare" entries that have some data and "commit" entries that say  
the data is no longer relevant.  I imagine you could write the ldif  
entry as the "prepare" record and some indication of all replicas  
being up-to-date as the "commit" entry.

activemq has quite a few journal/log schemes.  I think that they are  
mostly using kaha.  I'm don't understand all the features but think  
that it would do everything you need.  IIUC typically the messages  
themselves are written to one "database" and for each subscription an  
index entry is written to a related reference database.  As the acks  
come in for the subscriptions the index entries are removed and after  
all deliveries have happened the main entry is removed.

I imagine you could implement the whole replication scheme using  
activemq embedded brokers on each server but this might not be the  
most performant implementation.

david jencks

david jencks

>> wdyt ?
> thanks for putting the thoughts in ink, have a better understanding  
> now
> -- 
> Kiran Ayyagari

View raw message