Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: pass (athena.apache.org: domain of ayaselcuk@gmail.com
 designates 74.125.82.178 as permitted sender)
Received-SPF: pass (google.com: domain of ayaselcuk@gmail.com designates
 10.216.134.74 as permitted sender) client-ip=10.216.134.74;
MIME-Version: 1.0
In-Reply-To: <4F19A89F.101@gmail.com>
References: <4F19A89F.101@gmail.com>
Date: Sun, 19 Feb 2012 19:18:26 -0800
Message-ID: 
 <CALhXAG2Y_9Fa1kB2Xh1=iKjh6zOAtxKTfhpmMpqsk9ZekA-6_w@mail.gmail.com>
Subject: Re: Heads up
From: Selcuk AYA <ayaselcuk@gmail.com>
To: Apache Directory Developers List <dev@directory.apache.org>,
 elecharny@apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Jan 20, 2012 at 9:47 AM, Emmanuel Lecharny <elecharny@gmail.com> wr=
ote:
> Hi guys,
>
> I was a bit silent last week and this week, let me update you about what =
I
> was working on.
>
> First of all, I have had to deal with some familly issues, which ate half=
 of
> my time.
>
> Regarding the Txn branch I was working on until last week, I stopped beca=
use
> I was not able to fix the code without a serious help from Selcuk. As he =
is
> busy, I preferred to wait for him to be available again, instead of bully=
ing
> into the code and break it seriously. I believe that there are some
> improvement since the moment I started to work on the branch, but it's no=
t
> working fully yet.

I am starting to look into this again. I did an svn up and txn manager
tests in core.shared are not working. Do you have the same problem?
Also what was the test you thought was returning incorrect results? I
couldnt find the relevant email.


>
> So I switched to something we wanted to do a long time ago : designing a =
new
> version of JDBM. JDBM is a BTree implementation, with locks to protect
> concurrent access. The idea was to implement a MVCC solution on top of a
> BTree :
> - each search can be done concurrently with any other operation, because =
it
> asks for a specific existing revision from the btree
> - each modification is done on a new revision
> - two modifications can't be done at the same time (so modifications are
> queued and executed one after the other)
>
> The consequence is that searches will be very fast. It comes to a price
> though : we keep a track on every revision, until it's not used anymore.
> This is done by copying every modified pages when applying some
> modification.
>
> As of today, the addition operation and the find operation is working jus=
t
> fine. I conducted some benchmark on additions, and it seems that the syst=
em
> is pretty decent.
>
> A *lot* remains to be done :
> - deletion must be implemented
> - browsing the tree is not yet implemented
> - it's all in memory atm
> - we must add some semaphore for concurrent modifications
> - a GC must be added to discard unused pages
>
> But most of all, as it's a in-memory btree atm, I must add the disk layer=
.
> It will be based on Memory Mapped files.
>
> Once those preliminary works will be done, the idea is to use this
> implementation to replace JDBM. That would make the server consistent, an=
d
> we may then use it without the in-memory txn layer.
>
> Not to say that this txn layer is useless; using a MVCC btree based backe=
nd
> is *not* enough : we have no way to guarantee the atomicity of move
> operation across partitions.
>
> This work has been done in my sandbox, where you can follow the work in
> progress :
> http://svn.apache.org/viewvc/directory/sandbox/elecharny/shared-mvbt
>
> At the same time, thanks to Pierre-Arnaud, a first milstone of Studio 2.0
> has been released, and it exposed some nasty bugs in the LDAP API. Which =
is
> actually a good thing : we can fix them !
>
> So keep tuned, a lot of new things are coming soon !
>
> --
> Regards,
> Cordialement,
> Emmanuel L=E9charny
> www.iktek.com
>

thanks
Selcuk