From dev-return-57192-archive-asf-public=cust-asf.ponee.io@directory.apache.org Sun Feb 11 19:15:57 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id CC4DB18064E for ; Sun, 11 Feb 2018 19:15:57 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BC2B6160C4E; Sun, 11 Feb 2018 18:15:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0D1F3160C2E for ; Sun, 11 Feb 2018 19:15:56 +0100 (CET) Received: (qmail 26795 invoked by uid 500); 11 Feb 2018 18:15:56 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 26784 invoked by uid 99); 11 Feb 2018 18:15:56 -0000 Received: from ui-eu-01.ponee.io (HELO localhost) (176.9.59.70) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Feb 2018 18:15:56 +0000 Date: Sun, 11 Feb 2018 18:15:54 -0000 X-Mailer: LuaSocket 3.0-rc1 From: Emmanuel Lecharny Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 x-ponymail-agent: PonyMail Composer/0.2 References: Message-ID: Subject: Re: [ApacheDS] Transaction support In-Reply-To: x-ponymail-sender: 9d55c29c020f0607a252243a305e8fed47d50b37 To: Forgot ot mention that I tried to remove the Store interface, that was added a decade ago. I let it be, because it would have impacted the SchemaPartition. So we still have the Store interface around. Well... On 2018/02/11 17:43:33, Emmanuel Lécharny wrote: > Hi guys, > > a quick head up about this on-going effort. > > First of all, I added the Transaction extended operation in the LDAP > API, but this is somehow orthogonal. We don't really need that at the > moment in the server, but we will most certainly leverage it later for > some interesting feature (see later). > > At the moment, the idea is to add cross- B-tree transaction to > partitions (at least JDBM/Mavibot partitions). This is critical because > it will fix the corruption problem we have. > > The idea is to start a transaction in the OperationManager, either re ad > or write dependening on the operation. We can have many read operations > going on, but only one write operation (for the JDBM partition, we will > have some more constraints). > > Transactions have to be started by partitions, as the upper layer (ie > operation manager has no way to know how the lower level (ie partitions) > deal with transactions. This is possible because we can determinate > which partition we are addressing using the operation's DN. The first > thing to do is to move this part from partitions to the operation manager. > > Then we have to propagate the txn down to the nexus. The easier way is > to pass the txn into the OperationContext instances. Once we went > through the interceptors down to the Nexus partition, we have to apply > the operation to the specific partition, and this is done by the > AbstractBtreePartition, mostly. Enough said that each B-tree update > needs to know about the txn, so we have to modify the basic partition > operations to take an extra parameter : the txn. And this is where it > starts to be hard... Because that implies we also have to extend the > following interfaces : > > - Table > - Index > - Cursor > > so that they also take this txn as a parameter. > > It's a bit of a gigantic change in the interfaces... Note that we don't > want to change the LDAP API cursor interface, too. > > Then , we have to change teh way teh JDBM partition behave. Currently, > we create a RecordManager for each B-tree (ie, each Table, which may > have 2 B-tree, the forward and reverse index). This is not good, because > we can't apply a global operation across many recordManager using a txn. > The JDBM transactionManager is applied to a single RecordManager. So the > next big change is to have JDBM working using one single file. That > impacts the initialisation of JDBM index and partition. > > For mavibot, it's simpler, because we already use one single file anyway. > > ATM, I have made the required changes in the Partition/Index/Cursor to > pass this extrac txn paramer, and I'm dealing with the unique file > definition. The LDIF Partition is working just fine with all those > changes, and many of the JDBM tests are also running green. > > The last thing to do is for JDBM to make sure we don't have a collision > between reads and write (ie we can't read when we write). That will slow > down the server when we do a write, but JDBM won't be able to deal with > concurrent reads and writes anyway. Mavibot does not have such > limitation, so I expect that Mavibot will become teh de-facto backend > soon after those changes. > > LAst, not least, being able to leverage the Transaction extended > operation will allow a fast loading of the server, especially during the > initial injection of data: we can do that in memory, and flush teh > result globally. > > I expect all those changes to take a couple of weeks - working on > evenings and week-end). > > > > I'll keep you posted anyway. > > -- > Emmanuel Lecharny > > Symas.com > directory.apache.org > >