directory-kerby mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <elecha...@gmail.com>
Subject Re: Transaction support for Kerby backend
Date Thu, 24 Sep 2015 14:22:10 GMT
Le 21/09/15 13:29, Zheng, Kai a écrit :
> Hi all,
>
> This is proposing to add transaction support API for Kerby backend for efficiency. Kerby
provides various backends, some of them is file based, like the Json one. I'm attempting to
add another one based on Google Flatbuffers format. Such backends based on simple file would
be better to have transaction support for efficiency. In existing codes, every call to addIdentity/updateIdentity/deleteIdentity
will require to write the memory buffer/states to the disk file, quite inefficiently.
>
> For simple, it would be good enough to be:
>
> 1.       A backend instance allows only one transaction at a time;
>
> 2.       When it's in a transaction, any mutation operation via non-transaction API (existing
one) will be denied;
>
> 3.       In a transaction, multiple mutation operations can be made via the new transaction
API, and states are only updated to the memory, no store/save/flush to the disk file;
>
> 4.       When the transaction ends, the memory state will be persisted/synced to the
disk file, then the update content will be visible to other backend instances if it reloads.
>
> 5.       For backends that use a system already supporting transaction, like Mavibot,
LDAP and Zookeeper, the new transaction API will have default implementation that performs
no-op.
>
> I'd like to hold on some time for feedbacks before proceed, since I'm not expert for
such system designs. I wish it to be practical, simple, efficiency, and easy to do.
> Thanks.

So, let's talk about what is a transaction, in the kerby context.

Let's first talk about what is a Transaction in ApacheDS, just for the
record. In ApacheDS, when we update an entry, we impact many B-trees
(the Master table, which holds the entries, and the various indexes). We
currently don't have a cross-B-tree transaction, but we are working on
it. the idea is that either all the B-trees are updated as a whole, or
none of them are. That makes teh LDAP server consistent. It also implies
that reads aren't impacted by the writes.

In a Kerberos context, the problem is exactly the same, as many elements
might be modified when injecting some new element.

How I see the thing :
- if the underlying repository (or backend) does not support
transactions, then you are a bit in trouble. Typically, if the backend
is using flat files, then that means you have to copy the files when
updating them, which might be a costly operation, then swap the files
when done, and let the new readers using this new files, while the
existing readers keep going with the old files... Not simple !

JDBM has the exact same problem : it support transaction at the B-tree
level, but not across B-trees, which makes it  a wrong backend (as every
of us are realized 3 years ago :/) and this is the reason we started
Mavibot !

So I think kerby *NEEDS* to define a transaction layer on top of the
backends, but it also has to ensure than the backends support transactions.

Thoughts ?


Mime
View raw message