On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ayaselcuk@gmail.com> wrote:
Hi,
a few points and questions:
*I am planning to use a common ID for all partitions. I checked Hbase
partition and it uses UUID as the ID. Is it ok to use this for all
partitions?


This great and something we wanted to do for a very long time as Emmanuel stated.
 
*It seems that I need to move txn and log package implementations to
core-shared because of the recent reorg. Coresession and nexus
implementation is moved there and interceptors might need these
packages.


+1
 
* When preparing txn log edits, I will need to log Modification and
Attribute objects and keep them in memory for a while. Is it necessary
to clone these objects while doing the logging at partition nexus
level?

Don't think so.
 
In other words, will they be modified after reaching nexus
level. I am thinking no but I wanted to check because I looked at
changelog implementation and it does cloning.


Thanks,
Alex
 

On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <elecharny@gmail.com> wrote:
> Hi !
>
> comments inline...
>
> On 11/1/11 12:19 PM, Selcuk AYA wrote:
>>
>> thanks for the feedback! Please see inline:
>
> <snip/>
>>
>>
>> I guess the xdbm-partition Maven module is now going to be pretty damn
>> thin
>> or non-existent. Was this module destroyed and if not what actually
>> remains?
>> I think some util classes for tree based implementations remain there.
>> Also there some avl related classes.
>
> We can review this project and eventually move the remaining classes
> elsewhere.
>
>>
>>>> The todo is the following:
>>>> *add changes to keep track of dn space changes.
>>>> *test txn manager services
>>>> *move the modification code path in AbstractBTreePartition in xdbm to
>>>> high up to core. Probably modifications should be done in
>>>> partitionNexus and partition nexus should handle preparing txn log
>>>> edits and wal them. The overall flow for modifications will be:
>>>>   -DefaultDirectoryService:
>>>>          -begintxn
>>>>                            -execute interceptor chain
>>>>          - handle txn abort, or conflict.
>>>>
>>> Moving this into the PartitionNexus might not be a good idea but no
>>> problem
>>> for now we can move it later. Let me explain why:
>>> Eventually we're going to enable a root Partition with Partition nesting
>>> and
>>> so when this happens the PartitionNexus will just be another nestable
>>> Partition since these will have to handle routing based on DN to other
>>> partitions residing/nested under it.
>>> I see two possible locations for this functionality:
>>> (1) Let the InterceptorChain itself handle this since it can demarcate
>>> the
>>> start and end of calls into the chain with Txn begin and abort/commit
>>> calls.
>>> It does this by making calls against the TxnManager which I guess is the
>>> entire facade for the transaction subsystem.
>>> ---OR---
>>> (2) Handle Txn demarcation within the CoreSession. However this might not
>>> be
>>> optimal due to the need to handle additional logic which might be
>>> required
>>> for handling chain re-entry concerns.
>>> NOTE: I've not actually looked at the code after these major moves so my
>>> advice might not be very dependable. I will try to setup my environment
>>> to
>>> get a better idea of these matters.
>>> However for the time being do whatever actually makes this thing work.
>>> Let's
>>> follow an agile methodology. This thing is big. So let's get it working
>>> with
>>> solid test coverage then we can actually look at shuffling things around
>>> to
>>> optimal positions. Not saying what you've chosen is wrong ... it might
>>> just
>>> present the need for some additional refactoring when other features
>>> might
>>> need to be introduced.
>>> These are some of the biggest changes to the architecture to have taken
>>> place in years and you're doing a great job.
>>
>> You are right for txn demarcation. I wrote it wrong in the email.
>> Demarcation has to be done either at defaultcoresession or
>> defaultoperationmanager.
>
> OperationManager is probably the right place.
>
> For inner operations, I'm almost 100% sure they aren't modifiying anything
> (the only modification done as a inner operation has been removed a while
> ago : it was dealing with the addition of the ModifyTimeStamp and
> ModifiersName attribute, and it' snow done in the main operation). So we are
> safe if they are done in the current external transaction.
>
>>
>> as for where to handle the change logic, as you mentioned we need a
>> place where all interceptor chain routes end up for modification and
>> ideally we should handle modification logic above partitions using
>> master table and index interfaces so that we have a common place to
>> prepare and apply txn log edits. PartitionNexus seems to fit this
>> requirement for now. If we add another layer above PartitionNexus
>> which can get master and index table from below layers and work with
>> them, we should be able to move the change logic up there.
>>
>>>> *move xdbm-search to core as well. Making search transactional will
>>>> mostly be mostly mechanica after this point I think(hope). It should
>>>> just use the wrappers the txn manager provides for index, master and
>>>> cursors it gets from the partitions.
>>>>
>>>> *handle caches various interceptors keep. I am thinking of handling
>>>> this with a common read-write lock.
>>>>
>>> This was the latest issue for which I see some more threads. Will look at
>>> that as well.
>>
>> This is the issue I talked you about briefly. These are mostly admin
>> caches that change infrequently and that are read mostly. The
>> difficulty with them is that they are not always entry caches. They
>> might map Dns to some logical property of the entry. A simple and good
>> example is notaliascache (whether we really need this is open to
>> debate but we have it now).
>
> This is a very interesting point. The notAliasCache is definitively
> questionable. The question though is to find a better way to deal with it.
> An option would be to not support alias (yes, I know, pretty drastic, but if
> it allows us to speed up the delivery of a production ready server, I think
> this would be a good tradeoff. Also note that once we have a solid base, we
> can reintroduce alias handling in a future version).
>
>> This maps a Dn to whether the entry is an
>> alias. If we followed the normal way of merging what is read from
>> partitions with the txn log, we would have to a way of merging what is
>> read from this notaliascache with the log. This is not very difficult
>> but as we have quite a number of these caches, having a separate merge
>> and update logic for each of them is a pain and error prone.
>>
>> Instead, what I thought was to use a single system wide read-write
>> lock for these caches. Since they are read mostly, a txn might update
>> its lock to exclusive when it needs to change one of these caches and
>> might release it when it actually commits/aborts.
>
> We may start a new thread to discuss further those cache issues. We have
> many caches, for may different thinsg, and many different approaches to deal
> with the (MRU, EhCache, etc...). It's probably a good timing for clarifying
> this item...
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>



--
Best Regards,
-- Alex