directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <akaras...@apache.org>
Subject Re: TXN work update
Date Wed, 02 Nov 2011 10:15:39 GMT
On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ayaselcuk@gmail.com> wrote:

> Hi,
> a few points and questions:
> *I am planning to use a common ID for all partitions. I checked Hbase
> partition and it uses UUID as the ID. Is it ok to use this for all
> partitions?
>
>
This great and something we wanted to do for a very long time as Emmanuel
stated.


> *It seems that I need to move txn and log package implementations to
> core-shared because of the recent reorg. Coresession and nexus
> implementation is moved there and interceptors might need these
> packages.
>
>
+1


> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level?


Don't think so.


> In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
>
>
Thanks,
Alex


>
> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <elecharny@gmail.com>
> wrote:
> > Hi !
> >
> > comments inline...
> >
> > On 11/1/11 12:19 PM, Selcuk AYA wrote:
> >>
> >> thanks for the feedback! Please see inline:
> >
> > <snip/>
> >>
> >>
> >> I guess the xdbm-partition Maven module is now going to be pretty damn
> >> thin
> >> or non-existent. Was this module destroyed and if not what actually
> >> remains?
> >> I think some util classes for tree based implementations remain there.
> >> Also there some avl related classes.
> >
> > We can review this project and eventually move the remaining classes
> > elsewhere.
> >
> >>
> >>>> The todo is the following:
> >>>> *add changes to keep track of dn space changes.
> >>>> *test txn manager services
> >>>> *move the modification code path in AbstractBTreePartition in xdbm to
> >>>> high up to core. Probably modifications should be done in
> >>>> partitionNexus and partition nexus should handle preparing txn log
> >>>> edits and wal them. The overall flow for modifications will be:
> >>>>   -DefaultDirectoryService:
> >>>>          -begintxn
> >>>>                            -execute interceptor chain
> >>>>          - handle txn abort, or conflict.
> >>>>
> >>> Moving this into the PartitionNexus might not be a good idea but no
> >>> problem
> >>> for now we can move it later. Let me explain why:
> >>> Eventually we're going to enable a root Partition with Partition
> nesting
> >>> and
> >>> so when this happens the PartitionNexus will just be another nestable
> >>> Partition since these will have to handle routing based on DN to other
> >>> partitions residing/nested under it.
> >>> I see two possible locations for this functionality:
> >>> (1) Let the InterceptorChain itself handle this since it can demarcate
> >>> the
> >>> start and end of calls into the chain with Txn begin and abort/commit
> >>> calls.
> >>> It does this by making calls against the TxnManager which I guess is
> the
> >>> entire facade for the transaction subsystem.
> >>> ---OR---
> >>> (2) Handle Txn demarcation within the CoreSession. However this might
> not
> >>> be
> >>> optimal due to the need to handle additional logic which might be
> >>> required
> >>> for handling chain re-entry concerns.
> >>> NOTE: I've not actually looked at the code after these major moves so
> my
> >>> advice might not be very dependable. I will try to setup my environment
> >>> to
> >>> get a better idea of these matters.
> >>> However for the time being do whatever actually makes this thing work.
> >>> Let's
> >>> follow an agile methodology. This thing is big. So let's get it working
> >>> with
> >>> solid test coverage then we can actually look at shuffling things
> around
> >>> to
> >>> optimal positions. Not saying what you've chosen is wrong ... it might
> >>> just
> >>> present the need for some additional refactoring when other features
> >>> might
> >>> need to be introduced.
> >>> These are some of the biggest changes to the architecture to have taken
> >>> place in years and you're doing a great job.
> >>
> >> You are right for txn demarcation. I wrote it wrong in the email.
> >> Demarcation has to be done either at defaultcoresession or
> >> defaultoperationmanager.
> >
> > OperationManager is probably the right place.
> >
> > For inner operations, I'm almost 100% sure they aren't modifiying
> anything
> > (the only modification done as a inner operation has been removed a while
> > ago : it was dealing with the addition of the ModifyTimeStamp and
> > ModifiersName attribute, and it' snow done in the main operation). So we
> are
> > safe if they are done in the current external transaction.
> >
> >>
> >> as for where to handle the change logic, as you mentioned we need a
> >> place where all interceptor chain routes end up for modification and
> >> ideally we should handle modification logic above partitions using
> >> master table and index interfaces so that we have a common place to
> >> prepare and apply txn log edits. PartitionNexus seems to fit this
> >> requirement for now. If we add another layer above PartitionNexus
> >> which can get master and index table from below layers and work with
> >> them, we should be able to move the change logic up there.
> >>
> >>>> *move xdbm-search to core as well. Making search transactional will
> >>>> mostly be mostly mechanica after this point I think(hope). It should
> >>>> just use the wrappers the txn manager provides for index, master and
> >>>> cursors it gets from the partitions.
> >>>>
> >>>> *handle caches various interceptors keep. I am thinking of handling
> >>>> this with a common read-write lock.
> >>>>
> >>> This was the latest issue for which I see some more threads. Will look
> at
> >>> that as well.
> >>
> >> This is the issue I talked you about briefly. These are mostly admin
> >> caches that change infrequently and that are read mostly. The
> >> difficulty with them is that they are not always entry caches. They
> >> might map Dns to some logical property of the entry. A simple and good
> >> example is notaliascache (whether we really need this is open to
> >> debate but we have it now).
> >
> > This is a very interesting point. The notAliasCache is definitively
> > questionable. The question though is to find a better way to deal with
> it.
> > An option would be to not support alias (yes, I know, pretty drastic,
> but if
> > it allows us to speed up the delivery of a production ready server, I
> think
> > this would be a good tradeoff. Also note that once we have a solid base,
> we
> > can reintroduce alias handling in a future version).
> >
> >> This maps a Dn to whether the entry is an
> >> alias. If we followed the normal way of merging what is read from
> >> partitions with the txn log, we would have to a way of merging what is
> >> read from this notaliascache with the log. This is not very difficult
> >> but as we have quite a number of these caches, having a separate merge
> >> and update logic for each of them is a pain and error prone.
> >>
> >> Instead, what I thought was to use a single system wide read-write
> >> lock for these caches. Since they are read mostly, a txn might update
> >> its lock to exclusive when it needs to change one of these caches and
> >> might release it when it actually commits/aborts.
> >
> > We may start a new thread to discuss further those cache issues. We have
> > many caches, for may different thinsg, and many different approaches to
> deal
> > with the (MRU, EhCache, etc...). It's probably a good timing for
> clarifying
> > this item...
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>



-- 
Best Regards,
-- Alex

Mime
View raw message