directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <elecha...@gmail.com>
Subject Re: [Mavibot] Transaction support
Date Fri, 30 Oct 2015 09:14:27 GMT
Le 30/10/15 09:57, Howard Chu a écrit :
> Emmanuel Lécharny wrote:
>> Le 30/10/15 08:37, Howard Chu a écrit :
>>> Kiran Ayyagari wrote:
>>>>
>>>>
>>>> On Thu, Oct 29, 2015 at 6:13 PM, Emmanuel Lécharny
>>>> <elecharny@gmail.com
>>>> <mailto:elecharny@gmail.com>> wrote:
>>>>
>>>>      Hi guys,
>>>>
>>>>      here are a bit of my last night's insomnia thoughts (with the
>>>> shower,
>>>>      and the smallest room of my appartment, this is the only place
>>>> and time
>>>>      I have for such thoughts atm ;-) :
>>>>
>>>>      - txn are now going to be explicit, ie you have to begin a txn
>>>> and
>>>>      commit it before and after any update, like in :
>>>>
>>>>               // Inject some data
>>>>               recordManager.beginTransaction();
>>>>               btree.insert( 1L, "1" );
>>>>               btree.insert( 4L, "4" );
>>>>               btree.insert( 2L, "2" );
>>>>               btree.insert( 3L, "3" );
>>>>               btree.insert( 5L, "5" );
>>>>               recordManager.commit();
>>>>
>>>> if a user forgets to call commit() and keeps inserting we will run
>>>> out of
>>>> memory at some point
>>>> so wondering what is the better way to prevent that from happening
>>>> without
>>>> maintaining a
>>>> log (a WAL or something similar)
>>>
>>> In the original LMDB implementation we simply had a max number of
>>> dirty pages in a txn. Once that was hit, further write requests simply
>>> failed with MDB_TXN_FULL.
>>>
>>> In current versions we still have the same max number of dirty pages
>>> at once, but when the limit is hit we spill a portion of pages
>>> immediately to disk so we can reuse their memory. As such, the current
>>> version allows txns of unlimited size, but still with a fixed RAM
>>> footprint.
>>
>> This is ok if there is no relation between pages. The pb is that when
>> you use LMDB (or Mavibot) in a LDAP context, you will have to update
>> many pages in a single (application) transaction. Letting the underlying
>> DB to arbitrary flush pages based on a size/time limit simply break the
>> application transaction barriers.
>
> No, it doesn't break transaction barriers. Pages written to the DB are
> meaningless (and invisible to other DB users) until the final meta
> page update at the end of the transaction.

Yes, I see. But still, until the application developper actually close
the txn, the database remains in its previous state for a very long
time. This is perfectly ok, assuming the coder is not forgetting to do so...

OTOH, if a coder forget to explicitely close the txn, I can't see a use
case where he can't face the pb soon enough and package the application
with such a problem. So, bottom line, you are right : flushing the pages
when the limit is reached is a good enough protection against OOM.

>
>> We can think about another solution though : instead of having to
>> explicitely start a transaction, and commit it, we can imagine a backend
>> for which we pass the full list of operations to apply, up to the bakend
>> to play those operations and commit (or rollback) them all. This remind
>> me about 'continuation', where we computed the code to be executed,
>> before passing it to the part in charge of its execution (it was back in
>> 1988, when I was programing in LISP, where code is data).
>
> That doesn't change the problem at all. (In fact this is exactly how
> slapd implements LDAP txns. But it all boils down to somebody
> explicitly starting a txn, feeding in the sequence of ops, and
> explicitly committing at the end.)
The explicit commit is not anymore necessary in this scenario. Doing
things like : DB.apply( op1, oP2, ... opn) will start the txn and commit
it without requiring the app coder to take care of it at all.


Mime
View raw message