directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Howard Chu <...@symas.com>
Subject Re: Backend improvements
Date Sun, 17 Feb 2013 01:53:12 GMT
Emmanuel L├ęcharny wrote:
> 3) Switch to Mavibot
>
> Mavibot is a labs I started last year, which implements a MVCC Btree in
> memory. It's working, t's fast, but it's in memory only atm (although
> the data are regularly flushed on disk).
>
> We need to improve it so that we add those features :
> - Make it backed with a physical layer that access data on disk, and
> avoid storing everything in memory
> - Add a raw load feature
> - Add a global transaction system, cross trees.
>
> The last feature is probably easier to implement, as we do have a
> version associated with each new modification : if we retin the version
> we started with, we can gather all the btree's revision, and validate
> them all in one shot, or rollback them all in one shot (a revision is
> visible only if it has been validated).
>
> One very interestig aspect of Mavibot is that you can update the btrees
> concurrently with searches : searches are done on older revisions, which
> don't interract with the pages being added.
>
> I think there is one to two more month of work on Mavibot to get it to
> the level we need.
>
> Proposal
> --------
>
> IMO, we should investigate JDBM4 asap. It might solve many of the issues
> we have, at a low price. Improving the existing JDBM is just a waste of
> our precious time.
>
> I the long term, I would favor Mavibot for a couple of reasons :
> - first, it's a MVCC base, so we can conduct concurrent searches and
> modifications (although we can't do many modifications at the same time).
> - second, we have a complete understanding of the code : we wrote it.
> It's easier to understand the impact of the modifications we hve done on it.
> - third, we can have a dedicated backend based on the current Mavibot
> code, where all the data are in memory (that would work with a few tens
> of thousands entries, and the performances will be blazing fast).
>
> I'd like to know what's your opinion.

Is it a requirement to stick to pure java? We recently had someone inquire 
about a java wrapper for our Lightning DB library. Developing that could get 
you a big boost without as much development effort, assuming JNI is practical 
for deployments.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Mime
View raw message