directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@gmail.com>
Subject Heads up
Date Fri, 20 Jan 2012 17:47:11 GMT
Hi guys,

I was a bit silent last week and this week, let me update you about what 
I was working on.

First of all, I have had to deal with some familly issues, which ate 
half of my time.

Regarding the Txn branch I was working on until last week, I stopped 
because I was not able to fix the code without a serious help from 
Selcuk. As he is busy, I preferred to wait for him to be available 
again, instead of bullying into the code and break it seriously. I 
believe that there are some improvement since the moment I started to 
work on the branch, but it's not working fully yet.

So I switched to something we wanted to do a long time ago : designing a 
new version of JDBM. JDBM is a BTree implementation, with locks to 
protect concurrent access. The idea was to implement a MVCC solution on 
top of a BTree :
- each search can be done concurrently with any other operation, because 
it asks for a specific existing revision from the btree
- each modification is done on a new revision
- two modifications can't be done at the same time (so modifications are 
queued and executed one after the other)

The consequence is that searches will be very fast. It comes to a price 
though : we keep a track on every revision, until it's not used anymore. 
This is done by copying every modified pages when applying some 
modification.

As of today, the addition operation and the find operation is working 
just fine. I conducted some benchmark on additions, and it seems that 
the system is pretty decent.

A *lot* remains to be done :
- deletion must be implemented
- browsing the tree is not yet implemented
- it's all in memory atm
- we must add some semaphore for concurrent modifications
- a GC must be added to discard unused pages

But most of all, as it's a in-memory btree atm, I must add the disk 
layer. It will be based on Memory Mapped files.

Once those preliminary works will be done, the idea is to use this 
implementation to replace JDBM. That would make the server consistent, 
and we may then use it without the in-memory txn layer.

Not to say that this txn layer is useless; using a MVCC btree based 
backend is *not* enough : we have no way to guarantee the atomicity of 
move operation across partitions.

This work has been done in my sandbox, where you can follow the work in 
progress : 
http://svn.apache.org/viewvc/directory/sandbox/elecharny/shared-mvbt

At the same time, thanks to Pierre-Arnaud, a first milstone of Studio 
2.0 has been released, and it exposed some nasty bugs in the LDAP API. 
Which is actually a good thing : we can fix them !

So keep tuned, a lot of new things are coming soon !

-- 
Regards,
Cordialement,
Emmanuel L├ęcharny
www.iktek.com


Mime
View raw message