directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <akaras...@apache.org>
Subject Re: TXN work update
Date Tue, 01 Nov 2011 11:29:46 GMT
On Tue, Nov 1, 2011 at 1:19 PM, Selcuk AYA <ayaselcuk@gmail.com> wrote:

> thanks for the feedback! Please see inline:
>
> On Tue, Nov 1, 2011 at 5:35 AM, Alex Karasulu <akarasulu@apache.org>
> wrote:
> > On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ayaselcuk@gmail.com> wrote:
> >
> > Excuse the late response. Better late than never right ...
> > SNIP ...
> >
> >>
> >> *Master table, index and index cursor wrappers: These provide
> >> transactionally consistent views of entries and index data. They merge
> >> what they read from their wrapped objects with the txn log manager. To
> >> make this easier, txn manager keeps track of index changes for a
> >> read/write txn in a summary form. One thing to note is that not all
> >> information that wrappers expose is transactionall consistent. One
> >> example is count methods. No need to %100 accurate on this. For these
> >> cases, the wrapper just calls the wrapped object.
> >>
> >
> > Makes perfect sense to not bother with the count methods: slight
> variations
> > will not at all impact the optimizer's results.
> >
> >>
> >> *For all the above, interfaces are defined in core-api and
> >> implementations and internal interfaces are in core. Also moved,
> >> index, indexcursor, table and related interfaces and helper classes to
> >> core-api from xdbm-partition.
> >>
> >
> > I guess the xdbm-partition Maven module is now going to be pretty damn
> thin
> > or non-existent. Was this module destroyed and if not what actually
> remains?
>
> I think some util classes for tree based implementations remain there.
> Also there some avl related classes.
>
> >
> >>
> >> The todo is the following:
> >> *add changes to keep track of dn space changes.
> >> *test txn manager services
> >> *move the modification code path in AbstractBTreePartition in xdbm to
> >> high up to core. Probably modifications should be done in
> >> partitionNexus and partition nexus should handle preparing txn log
> >> edits and wal them. The overall flow for modifications will be:
> >>   -DefaultDirectoryService:
> >>          -begintxn
> >>                            -execute interceptor chain
> >>          - handle txn abort, or conflict.
> >>
> >
> > Moving this into the PartitionNexus might not be a good idea but no
> problem
> > for now we can move it later. Let me explain why:
> > Eventually we're going to enable a root Partition with Partition nesting
> and
> > so when this happens the PartitionNexus will just be another nestable
> > Partition since these will have to handle routing based on DN to other
> > partitions residing/nested under it.
> > I see two possible locations for this functionality:
> > (1) Let the InterceptorChain itself handle this since it can demarcate
> the
> > start and end of calls into the chain with Txn begin and abort/commit
> calls.
> > It does this by making calls against the TxnManager which I guess is the
> > entire facade for the transaction subsystem.
> > ---OR---
> > (2) Handle Txn demarcation within the CoreSession. However this might
> not be
> > optimal due to the need to handle additional logic which might be
> required
> > for handling chain re-entry concerns.
> > NOTE: I've not actually looked at the code after these major moves so my
> > advice might not be very dependable. I will try to setup my environment
> to
> > get a better idea of these matters.
> > However for the time being do whatever actually makes this thing work.
> Let's
> > follow an agile methodology. This thing is big. So let's get it working
> with
> > solid test coverage then we can actually look at shuffling things around
> to
> > optimal positions. Not saying what you've chosen is wrong ... it might
> just
> > present the need for some additional refactoring when other features
> might
> > need to be introduced.
> > These are some of the biggest changes to the architecture to have taken
> > place in years and you're doing a great job.
>
> You are right for txn demarcation. I wrote it wrong in the email.
> Demarcation has to be done either at defaultcoresession or
> defaultoperationmanager.
>
> as for where to handle the change logic, as you mentioned we need a
> place where all interceptor chain routes end up for modification and
> ideally we should handle modification logic above partitions using
> master table and index interfaces so that we have a common place to
> prepare and apply txn log edits. PartitionNexus seems to fit this
> requirement for now. If we add another layer above PartitionNexus
> which can get master and index table from below layers and work with
> them, we should be able to move the change logic up there.
>
> >>
> >> *move xdbm-search to core as well. Making search transactional will
> >> mostly be mostly mechanica after this point I think(hope). It should
> >> just use the wrappers the txn manager provides for index, master and
> >> cursors it gets from the partitions.
> >>
> >> *handle caches various interceptors keep. I am thinking of handling
> >> this with a common read-write lock.
> >>
> >
> > This was the latest issue for which I see some more threads. Will look at
> > that as well.
> This is the issue I talked you about briefly. These are mostly admin
> caches that change infrequently and that are read mostly. The
> difficulty with them is that they are not always entry caches. They
> might map Dns to some logical property of the entry. A simple and good
> example is notaliascache (whether we really need this is open to
> debate but we have it now). This maps a Dn to whether the entry is an
> alias. If we followed the normal way of merging what is read from
> partitions with the txn log, we would have to a way of merging what is
> read from this notaliascache with the log. This is not very difficult
> but as we have quite a number of these caches, having a separate merge
> and update logic for each of them is a pain and error prone.
>
> Instead, what I thought was to use a single system wide read-write
> lock for these caches. Since they are read mostly, a txn might update
> its lock to exclusive when it needs to change one of these caches and
> might release it when it actually commits/aborts.
>
>
Yep I remember this conversation very well. I think this is the best route
for us to take for the time being. Handling each cache separate takes more
effort and it might not pay off so let's not do unnecessary
write-ahead-coding :-D.

Who knows it might be the best way for us to proceed from all aspects
including performance. You just can't tell these things until you test the
server in a stress test environment. That's why we need a solid test
environment.

-- 
Best Regards,
-- Alex

Mime
View raw message