directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Howard Chu <...@symas.com>
Subject Re: [Mavibot] Transaction support
Date Fri, 30 Oct 2015 08:57:04 GMT
Emmanuel Lécharny wrote:
> Le 30/10/15 08:37, Howard Chu a écrit :
>> Kiran Ayyagari wrote:
>>>
>>>
>>> On Thu, Oct 29, 2015 at 6:13 PM, Emmanuel Lécharny <elecharny@gmail.com
>>> <mailto:elecharny@gmail.com>> wrote:
>>>
>>>      Hi guys,
>>>
>>>      here are a bit of my last night's insomnia thoughts (with the
>>> shower,
>>>      and the smallest room of my appartment, this is the only place
>>> and time
>>>      I have for such thoughts atm ;-) :
>>>
>>>      - txn are now going to be explicit, ie you have to begin a txn and
>>>      commit it before and after any update, like in :
>>>
>>>               // Inject some data
>>>               recordManager.beginTransaction();
>>>               btree.insert( 1L, "1" );
>>>               btree.insert( 4L, "4" );
>>>               btree.insert( 2L, "2" );
>>>               btree.insert( 3L, "3" );
>>>               btree.insert( 5L, "5" );
>>>               recordManager.commit();
>>>
>>> if a user forgets to call commit() and keeps inserting we will run
>>> out of
>>> memory at some point
>>> so wondering what is the better way to prevent that from happening
>>> without
>>> maintaining a
>>> log (a WAL or something similar)
>>
>> In the original LMDB implementation we simply had a max number of
>> dirty pages in a txn. Once that was hit, further write requests simply
>> failed with MDB_TXN_FULL.
>>
>> In current versions we still have the same max number of dirty pages
>> at once, but when the limit is hit we spill a portion of pages
>> immediately to disk so we can reuse their memory. As such, the current
>> version allows txns of unlimited size, but still with a fixed RAM
>> footprint.
>
> This is ok if there is no relation between pages. The pb is that when
> you use LMDB (or Mavibot) in a LDAP context, you will have to update
> many pages in a single (application) transaction. Letting the underlying
> DB to arbitrary flush pages based on a size/time limit simply break the
> application transaction barriers.

No, it doesn't break transaction barriers. Pages written to the DB are 
meaningless (and invisible to other DB users) until the final meta page update 
at the end of the transaction.

> We can think about another solution though : instead of having to
> explicitely start a transaction, and commit it, we can imagine a backend
> for which we pass the full list of operations to apply, up to the bakend
> to play those operations and commit (or rollback) them all. This remind
> me about 'continuation', where we computed the code to be executed,
> before passing it to the part in charge of its execution (it was back in
> 1988, when I was programing in LISP, where code is data).

That doesn't change the problem at all. (In fact this is exactly how slapd 
implements LDAP txns. But it all boils down to somebody explicitly starting a 
txn, feeding in the sequence of ops, and explicitly committing at the end.)

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Mime
View raw message