directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <>
Subject Re: TXN WORK: advice needed on how to deail with logical caches
Date Mon, 17 Oct 2011 16:12:37 GMT
On 10/17/11 3:31 PM, Selcuk AYA wrote:
> Hi all,
> I am hoping to send a more detailed email as to where I am in the txn
> implementation but I noticed I might hit a snag with regard to the
> various logical caches we maintain.
Ahha... Caches... Pain...

We have many of them. Let's first list all the caches we are using :
- dnCache : a DN cache used to spare DN parsing. If the DN is not 
anymore valid, it should be removed from the cache.
- subentryCache : a DN -> Subentry Map. We should update it when a 
subentry is added/modified
- accessControlXXX caches : Caches used for AccessPoint. It's a DnNode 
data structure, using the DN as an entry point.
- groupCache : a cache containing the groups
- tupleCache : a cach containing the ACI tuples
- kdcReplayCache : a Kerberos cache
- referrals cache : a cache used to manage referrals
- credentialCache : a LRUMap used for authentication
- registrations : a cache of notifications
- ObjctClass chaches (must, may, superiors, allowed)
- TriggerSpecCache A cache of id for the Triggers
- notAliasCache : This is a weird alias. It's used to know if an entry's 
parent is not an alias. IMO, we should rather have an Alias cache...

This is pretty much all the cache we declare, if we exclude the index 
and master table cache (entry).

Obviously, many of those caches will be impacted by any modification 
done in the server.
> Emmanuel is moving these caches out of interceptor but up until now
> these caches were in interceptors. They map entry DN to a logical
> value that is a predicate of the entry attributes. Currently I am
> aware of notAlias and subentry cache(there could be more) as such
> caches but it is not difficult to see people might add such cache in
> their custom interceptors.
> An example of the transactional execution we might have according to
> the planned implementation is this:
> R1, T1, T2 ->  R1 is before T1 and T2 and should be isolated from them.
> T1 is committed and T2 started after T1, so it should see the affects
> of T1. Also changes of T1 are not reflected(flushed) to the underlying
> partitions yet. Remeber that readers merge what they read from
> partitions with the changes in the unflushed part of the txn log.
> Now considers how we would make R1 and T2 see a consistent state of
> notAlias and subentry cache. It seems to me the only possible way  is
> to go ahead like we do with entries and index values: Update the cache
> when the txn log is being flushed to the partitions  and when these
> caches read, merge whatever we read with the txn log. However, each
> separate cache requires a separate logic to handle this merge and i am
> afraid it might complicated and slow as the number of such caches
> increase. Especially expecting a custom cache implementer to get this
> right seems dubious.
> please let me know what you think

This is plain right. We currently don't handle any kind of transaction 
system for cache, so we might very well end with a out of date cache at 
some point. This is dangerous... OTOH, as we now are implementing a MVCC 
mechanism, keeping the cach as they are is just not an option, and we 
must keep the revision for entries and DN, as they might have been 
changed by another thread...

This is not an easy issue.

In many places, we are now using ehCache to manage caches, instead of 
using a LRUMap, for instance, but this is not the case for all the 
caches. For instance, in some places, we are using a DnNode cache (which 
s used for partitions, accessControl, etc).

One possibility would be to associate the revision to each key, assuming 
that each operation will have a revision number in its context. Any 
modification impacting any cache will just create a new element with a 
revision into those cache.

In order to improve the cache management, it would be good to always use 
the CacheService, which provides some methods to easily manage caches. 
That may be possible except for the DnNode cache, as it's a tree 

We should think seriously about the best way to solve this issue, as 
it's really critical.

Thanks Selcuk !
> regards
> Selcuk

Emmanuel L├ęcharny

View raw message