Mailing-List: contact kerby-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kerby@directory.apache.org
Subject: Re: Transaction support for Kerby backend
To: kerby@directory.apache.org
References: 
 <8D5F7E3237B3ED47B84CF187BB17B66611C43363@SHSMSX103.ccr.corp.intel.com>
From: =?UTF-8?Q?Emmanuel_L=c3=a9charny?= <elecharny@gmail.com>
Message-ID: <56040712.2050205@gmail.com>
Date: Thu, 24 Sep 2015 16:22:10 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0)
 Gecko/20100101 Thunderbird/38.2.0
MIME-Version: 1.0
In-Reply-To: 
 <8D5F7E3237B3ED47B84CF187BB17B66611C43363@SHSMSX103.ccr.corp.intel.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Le 21/09/15 13:29, Zheng, Kai a =C3=A9crit :
> Hi all,
>
> This is proposing to add transaction support API for Kerby backend for =
efficiency. Kerby provides various backends, some of them is file based, =
like the Json one. I'm attempting to add another one based on Google Flat=
buffers format. Such backends based on simple file would be better to hav=
e transaction support for efficiency. In existing codes, every call to ad=
dIdentity/updateIdentity/deleteIdentity will require to write the memory =
buffer/states to the disk file, quite inefficiently.
>
> For simple, it would be good enough to be:
>
> 1.       A backend instance allows only one transaction at a time;
>
> 2.       When it's in a transaction, any mutation operation via non-tra=
nsaction API (existing one) will be denied;
>
> 3.       In a transaction, multiple mutation operations can be made via=
 the new transaction API, and states are only updated to the memory, no s=
tore/save/flush to the disk file;
>
> 4.       When the transaction ends, the memory state will be persisted/=
synced to the disk file, then the update content will be visible to other=
 backend instances if it reloads.
>
> 5.       For backends that use a system already supporting transaction,=
 like Mavibot, LDAP and Zookeeper, the new transaction API will have defa=
ult implementation that performs no-op.
>
> I'd like to hold on some time for feedbacks before proceed, since I'm n=
ot expert for such system designs. I wish it to be practical, simple, eff=
iciency, and easy to do.
> Thanks.

So, let's talk about what is a transaction, in the kerby context.

Let's first talk about what is a Transaction in ApacheDS, just for the
record. In ApacheDS, when we update an entry, we impact many B-trees
(the Master table, which holds the entries, and the various indexes). We
currently don't have a cross-B-tree transaction, but we are working on
it. the idea is that either all the B-trees are updated as a whole, or
none of them are. That makes teh LDAP server consistent. It also implies
that reads aren't impacted by the writes.

In a Kerberos context, the problem is exactly the same, as many elements
might be modified when injecting some new element.

How I see the thing :
- if the underlying repository (or backend) does not support
transactions, then you are a bit in trouble. Typically, if the backend
is using flat files, then that means you have to copy the files when
updating them, which might be a costly operation, then swap the files
when done, and let the new readers using this new files, while the
existing readers keep going with the old files... Not simple !

JDBM has the exact same problem : it support transaction at the B-tree
level, but not across B-trees, which makes it  a wrong backend (as every
of us are realized 3 years ago :/) and this is the reason we started
Mavibot !

So I think kerby *NEEDS* to define a transaction layer on top of the
backends, but it also has to ensure than the backends support transaction=
s.

Thoughts ?