Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: pass (athena.apache.org: domain of hyc@symas.com designates
 64.71.152.235 as permitted sender)
Message-ID: <4F1AE8D9.5040405@symas.com>
Date: Sat, 21 Jan 2012 08:33:29 -0800
From: Howard Chu <hyc@symas.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:12.0a1) Gecko/20111224 Firefox/12.0a1 SeaMonkey/2.9a1
MIME-Version: 1.0
To: Apache Directory Developers List <dev@directory.apache.org>
CC: Emmanuel Lecharny <elecharny@gmail.com>
Subject: Re: Heads up
References: <4F19A89F.101@gmail.com>
In-Reply-To: <4F19A89F.101@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Emmanuel Lecharny wrote:
> Hi guys,
>
> I was a bit silent last week and this week, let me update you about what
> I was working on.

Sounds very cool!
>
> First of all, I have had to deal with some familly issues, which ate
> half of my time.
>
> Regarding the Txn branch I was working on until last week, I stopped
> because I was not able to fix the code without a serious help from
> Selcuk. As he is busy, I preferred to wait for him to be available
> again, instead of bullying into the code and break it seriously. I
> believe that there are some improvement since the moment I started to
> work on the branch, but it's not working fully yet.
>
> So I switched to something we wanted to do a long time ago : designing a
> new version of JDBM. JDBM is a BTree implementation, with locks to
> protect concurrent access. The idea was to implement a MVCC solution on
> top of a BTree :
> - each search can be done concurrently with any other operation, because
> it asks for a specific existing revision from the btree
> - each modification is done on a new revision
> - two modifications can't be done at the same time (so modifications are
> queued and executed one after the other)
>
> The consequence is that searches will be very fast. It comes to a price
> though : we keep a track on every revision, until it's not used anymore.
> This is done by copying every modified pages when applying some
> modification.
>
> As of today, the addition operation and the find operation is working
> just fine. I conducted some benchmark on additions, and it seems that
> the system is pretty decent.
>
> A *lot* remains to be done :
> - deletion must be implemented
> - browsing the tree is not yet implemented
> - it's all in memory atm
> - we must add some semaphore for concurrent modifications
> - a GC must be added to discard unused pages
>
> But most of all, as it's a in-memory btree atm, I must add the disk
> layer. It will be based on Memory Mapped files.
>
> Once those preliminary works will be done, the idea is to use this
> implementation to replace JDBM. That would make the server consistent,
> and we may then use it without the in-memory txn layer.
>
> Not to say that this txn layer is useless; using a MVCC btree based
> backend is *not* enough : we have no way to guarantee the atomicity of
> move operation across partitions.
>
> This work has been done in my sandbox, where you can follow the work in
> progress :
> http://svn.apache.org/viewvc/directory/sandbox/elecharny/shared-mvbt
>
> At the same time, thanks to Pierre-Arnaud, a first milstone of Studio
> 2.0 has been released, and it exposed some nasty bugs in the LDAP API.
> Which is actually a good thing : we can fix them !
>
> So keep tuned, a lot of new things are coming soon !
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/