Emmanuel,

On Fri, Sep 26, 2008 at 2:01 PM, Emmanuel Lecharny <elecharny@gmail.com> wrote:
Hi guys,

as i'm struggling with referrals for days now, I just discovered that there are many places in the server which are not handling them correctly.

I had removed referral handling from everywhere and placed it only at the protocol provider layer where it does work properly.
 
Atm, the only part which is sane is the protocol layer. For embedded servers, not passing through the protocol layer, I have to say that the referrals are simply ignored.

This statement is not completely accurate.  When you embed ApacheDS as a server with the protocol layer then the protocol layer will handle referrals.  You probably mean to say the DirectoryService and the JNDI Provider around the DirectoryService does not support referrals.
 
Forgive me if I sound like I'm trying to correct you but just trying to make sure we use the right terminology to note exactly what we're talking about.


Here are the problem we have.

A bit of history. Before we removed JNDI from the server, here was the chain followed by an incoming request :

1) through the network
client --> JNDI--> {network}--> ASN.1codec --> Protocol handler --> JNDI --> NexusProxy --> Interceptors --> ...

2) embedded
client --> JNDI--> NexusProxy --> Interceptors --> ...

Simple, clear, but messy inside the server.

We decided to remove JNDI from the server, to offer our own API (CoreSession) on top of which we were supposed to rewrite the JNDI layer. here is what are suposed to have now :
1) through the network
client --> JNDI--> {network}--> ASN.1codec --> Protocol handler --> CoreSession --> OperationManager --> Interceptors ...

2) embedded with JNDI
client --> JNDI--> CoreSession --> OperationManager --> Interceptors ...

3) embedded without JNDI
client --> CoreSession --> OperationManager --> Interceptors ...

As everyone can see, the JNDI provider is just offered as a thin layer to the users who don't want to deal with ADS inside out.

Right. Users who do not want to deal with the internal API's can just initialize the DirectoryService and use it to create an InitialContext.  There after all operations appear to be remote through JNDI which users are used to.
 
This is perfect for those who want to migrate from version X to version Y. But they can also use the better API whoch does not use the cursed semantic of JNDI.

So far, so good.

Ahha.
 

Now, the problem is that for referrals, as we have also removed the ReferralInterceptor from the core server (a very good move, when you take the time to think about it - and it took me two hours of discussion with Alex to get it -), you have a new problem : where do we have to handle referrals ?

Question is for what purpose do you want to handle referrals.  There are two ways to handle referrals:

(1) Through the protocol
(2) Via the JNDI Provided based on how JNDI is expected to behave with different values for the Context.REFERRAL property in the environment: throw, ignore, follow.
 

The first idea was to move it to the protocol layer. So be it. But the problem is that the JNDI referral handling was not anymore done (and as a consequence, the associated tests were disabled, until the JNDI provider was fixed to handle those referrals).

I'm currently re-implementing the Referral handling in this JNDI layer, but I suddenly discovered that this was not a good idea *at all*. And so was the initial idea to implement the referrals handling into the protocol layer. We now have 3 places where we must implement this process :
- in the protocol layer
- in the JNDI provider
- and in the CoreSession layer.

Bad...

I don't think it should be done in the CoreSession.  The CoreSession should just provide an interface to perform operations against entries.  I think all entries should be treated the same with respect to this interface.

If code needs to do something different because the entry is a referral it's up to that code to do so. 

Referral handling is different depending on whether the protocol is handling it or whether the JNDI provider is handling it.  The reason why I separated these is because the concepts were getting mixed together and JNDI was contaminating the server.


So here is what I suggest : we should handle referrals into the CoreSession layer, and deal with the thrown exceptions (LdapReferralExceptio, LdapPartialResultException, etc) into the JNDI and protocol layer, otherwise, we will do the job three times (and it's not a funny job ...)

thoughts ?

We could do this but these exceptions are JNDI specific.  They've been defined to deal with a specific JNDI referral handling mode (throw specifically).  Then the protocol handler code would need to change to understand these JNDI specifics.

Perhaps we should just not deal with referrals at all in the CoreSession.  Just add it to the JNDI provider for it's specific modes.

My mindset my be wrong (and at times I do doubt it) but I feel the core API is purely a local access API to initialize and manipulate the DIT and its entries.  It does not deal with following referrals since this would go over the network.  It does not deal with referrals as special entries since this is the responsibility of higher layers which deal with referrals in different ways. 
 

PS: don't be too picky with the names I'm using (like CoreSession is not really the API we have built), but I'm sure that Alex can correct me. I think however that the general picture is ok.


Yeah just great - I understood easily what you meant.  I think reuse is good and I can certainly see how putting some referral handling functionality into the CoreSession is very tempting.  As coders we always want to reuse and that's good.  However sometimes reuse causes more problems than it is worth.  I think this is a good example for that kind of situation.  Also sometimes we want to add additional functionality because it seems to make sense to have a richer interface that handles more cases.  However more complex interfaces are harder to work with as bricks to build other interfaces when their rich features start to get in the way.

I trust you know the best way to proceed and you'll make it work well.  I still however prefer the core's job to be as simple as possible.  Perhaps it can go just a little bit further and flag Entries as being referrals so higher layers can easily see that they're dealing with a referral entry.  The entry API can be altered to check if an entry is a referral with an isReferral() utility method which is non-essential since we can just check if the 'ref' attribute is present.  It might be cleaner though.  If this is done you might also add a flag indicating that the entry is subordinate to another referral.  This information is available through the ReferralManager and can be accessed to set this flag before returning an entry.  Perhaps this method can be called getFarthestReferralParent() which returns an LdapDN instead of just a boolean check.  These additions make it easier to use the Entry API without having to access the DirectoryService's ReferralManager to check for these.  Then the referral handling code, whether in the protocol provider or in the JNDI provider, has a much easier task.  The respective code is clean and shows clearly the policy of how referrals are handled in their respective layers.

Regards,
Alex