directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Selcuk AYA <ayasel...@gmail.com>
Subject Re: TXN work update
Date Thu, 03 Nov 2011 15:44:50 GMT
On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu <akarasulu@apache.org> wrote:
> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ayaselcuk@gmail.com> wrote:
>>
>> Hi,
>> a few points and questions:
>> *I am planning to use a common ID for all partitions. I checked Hbase
>> partition and it uses UUID as the ID. Is it ok to use this for all
>> partitions?
>>
>
> This great and something we wanted to do for a very long time as Emmanuel
> stated.
I am working on JDBM and AVL partitions to make them usee UUID. Also I
am making partitions expose their master tabe and indices. For making
all partitions use UUID:
* I can  change partitions to work with UUID directly  and make them
return master tables as MasterTable<Entry,UUID> and index tables in a
similar way
*or I can paratmetrize partition interface like Partition<ID> . They
can return master table as <MasterTable,ID> and index tables in a
similar way. This is looks to me the right way but it eventually leads
to more changes.

which one do you think is better?


>
>>
>> *It seems that I need to move txn and log package implementations to
>> core-shared because of the recent reorg. Coresession and nexus
>> implementation is moved there and interceptors might need these
>> packages.
>>
>
> +1
>
>>
>> * When preparing txn log edits, I will need to log Modification and
>> Attribute objects and keep them in memory for a while. Is it necessary
>> to clone these objects while doing the logging at partition nexus
>> level?
>
> Don't think so.
>
>>
>> In other words, will they be modified after reaching nexus
>> level. I am thinking no but I wanted to check because I looked at
>> changelog implementation and it does cloning.
>>
>
> Thanks,
> Alex
>
>>
>> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <elecharny@gmail.com>
>> wrote:
>> > Hi !
>> >
>> > comments inline...
>> >
>> > On 11/1/11 12:19 PM, Selcuk AYA wrote:
>> >>
>> >> thanks for the feedback! Please see inline:
>> >
>> > <snip/>
>> >>
>> >>
>> >> I guess the xdbm-partition Maven module is now going to be pretty damn
>> >> thin
>> >> or non-existent. Was this module destroyed and if not what actually
>> >> remains?
>> >> I think some util classes for tree based implementations remain there.
>> >> Also there some avl related classes.
>> >
>> > We can review this project and eventually move the remaining classes
>> > elsewhere.
>> >
>> >>
>> >>>> The todo is the following:
>> >>>> *add changes to keep track of dn space changes.
>> >>>> *test txn manager services
>> >>>> *move the modification code path in AbstractBTreePartition in xdbm
to
>> >>>> high up to core. Probably modifications should be done in
>> >>>> partitionNexus and partition nexus should handle preparing txn log
>> >>>> edits and wal them. The overall flow for modifications will be:
>> >>>>   -DefaultDirectoryService:
>> >>>>          -begintxn
>> >>>>                            -execute interceptor chain
>> >>>>          - handle txn abort, or conflict.
>> >>>>
>> >>> Moving this into the PartitionNexus might not be a good idea but no
>> >>> problem
>> >>> for now we can move it later. Let me explain why:
>> >>> Eventually we're going to enable a root Partition with Partition
>> >>> nesting
>> >>> and
>> >>> so when this happens the PartitionNexus will just be another nestable
>> >>> Partition since these will have to handle routing based on DN to other
>> >>> partitions residing/nested under it.
>> >>> I see two possible locations for this functionality:
>> >>> (1) Let the InterceptorChain itself handle this since it can demarcate
>> >>> the
>> >>> start and end of calls into the chain with Txn begin and abort/commit
>> >>> calls.
>> >>> It does this by making calls against the TxnManager which I guess is
>> >>> the
>> >>> entire facade for the transaction subsystem.
>> >>> ---OR---
>> >>> (2) Handle Txn demarcation within the CoreSession. However this might
>> >>> not
>> >>> be
>> >>> optimal due to the need to handle additional logic which might be
>> >>> required
>> >>> for handling chain re-entry concerns.
>> >>> NOTE: I've not actually looked at the code after these major moves so
>> >>> my
>> >>> advice might not be very dependable. I will try to setup my
>> >>> environment
>> >>> to
>> >>> get a better idea of these matters.
>> >>> However for the time being do whatever actually makes this thing work.
>> >>> Let's
>> >>> follow an agile methodology. This thing is big. So let's get it
>> >>> working
>> >>> with
>> >>> solid test coverage then we can actually look at shuffling things
>> >>> around
>> >>> to
>> >>> optimal positions. Not saying what you've chosen is wrong ... it might
>> >>> just
>> >>> present the need for some additional refactoring when other features
>> >>> might
>> >>> need to be introduced.
>> >>> These are some of the biggest changes to the architecture to have
>> >>> taken
>> >>> place in years and you're doing a great job.
>> >>
>> >> You are right for txn demarcation. I wrote it wrong in the email.
>> >> Demarcation has to be done either at defaultcoresession or
>> >> defaultoperationmanager.
>> >
>> > OperationManager is probably the right place.
>> >
>> > For inner operations, I'm almost 100% sure they aren't modifiying
>> > anything
>> > (the only modification done as a inner operation has been removed a
>> > while
>> > ago : it was dealing with the addition of the ModifyTimeStamp and
>> > ModifiersName attribute, and it' snow done in the main operation). So we
>> > are
>> > safe if they are done in the current external transaction.
>> >
>> >>
>> >> as for where to handle the change logic, as you mentioned we need a
>> >> place where all interceptor chain routes end up for modification and
>> >> ideally we should handle modification logic above partitions using
>> >> master table and index interfaces so that we have a common place to
>> >> prepare and apply txn log edits. PartitionNexus seems to fit this
>> >> requirement for now. If we add another layer above PartitionNexus
>> >> which can get master and index table from below layers and work with
>> >> them, we should be able to move the change logic up there.
>> >>
>> >>>> *move xdbm-search to core as well. Making search transactional will
>> >>>> mostly be mostly mechanica after this point I think(hope). It should
>> >>>> just use the wrappers the txn manager provides for index, master
and
>> >>>> cursors it gets from the partitions.
>> >>>>
>> >>>> *handle caches various interceptors keep. I am thinking of handling
>> >>>> this with a common read-write lock.
>> >>>>
>> >>> This was the latest issue for which I see some more threads. Will look
>> >>> at
>> >>> that as well.
>> >>
>> >> This is the issue I talked you about briefly. These are mostly admin
>> >> caches that change infrequently and that are read mostly. The
>> >> difficulty with them is that they are not always entry caches. They
>> >> might map Dns to some logical property of the entry. A simple and good
>> >> example is notaliascache (whether we really need this is open to
>> >> debate but we have it now).
>> >
>> > This is a very interesting point. The notAliasCache is definitively
>> > questionable. The question though is to find a better way to deal with
>> > it.
>> > An option would be to not support alias (yes, I know, pretty drastic,
>> > but if
>> > it allows us to speed up the delivery of a production ready server, I
>> > think
>> > this would be a good tradeoff. Also note that once we have a solid base,
>> > we
>> > can reintroduce alias handling in a future version).
>> >
>> >> This maps a Dn to whether the entry is an
>> >> alias. If we followed the normal way of merging what is read from
>> >> partitions with the txn log, we would have to a way of merging what is
>> >> read from this notaliascache with the log. This is not very difficult
>> >> but as we have quite a number of these caches, having a separate merge
>> >> and update logic for each of them is a pain and error prone.
>> >>
>> >> Instead, what I thought was to use a single system wide read-write
>> >> lock for these caches. Since they are read mostly, a txn might update
>> >> its lock to exclusive when it needs to change one of these caches and
>> >> might release it when it actually commits/aborts.
>> >
>> > We may start a new thread to discuss further those cache issues. We have
>> > many caches, for may different thinsg, and many different approaches to
>> > deal
>> > with the (MRU, EhCache, etc...). It's probably a good timing for
>> > clarifying
>> > this item...
>> >
>> > --
>> > Regards,
>> > Cordialement,
>> > Emmanuel Lécharny
>> > www.iktek.com
>> >
>> >
>
>
>
> --
> Best Regards,
> -- Alex
>

Mime
View raw message