Hi Emmanuel,

On Sat, Dec 27, 2008 at 7:42 PM, Emmanuel Lecharny <elecharny@gmail.com> wrote:
Hi guys,

as I'm reviewing the configuration, and as I want to cover the LdapService class, I'm a bit concerned with the current hierarchy of class we are using. Let me explain :

We have two categories of elements : Services and Servers.

Services : Ldap and Ldaps, plus a DirectoryBackedService
Servers  : ApacheDS, KdcServer, DnsServer, NtpServer, ChangePasswordServer

Yes you're right this categorization is flawed and confusing. The history of this faulty evolution was due to a merging of ideas from Enrique and myself.  I tried to align it and make it consistent over time but my attempts were interrupted.

The current inheritence scheme is :

(ProtocolService) o--- [AbstractProtocolService] <-- [NtpServer]
(ProtocolService) o--- [AbstractProtocolService] <-- [DirectoryBackedService] <-- [LdapService]

These seem consistent.

(ProtocolService) o--- [AbstractProtocolService] <-- [DirectoryBackedService] <-- [KdcServer]
(ProtocolService) o--- [AbstractProtocolService] <-- [DirectoryBackedService] <-- [DnsServer]
(ProtocolService) o--- [AbstractProtocolService] <-- [DirectoryBackedService] <-- [ChangePasswordServer]

Here I think we just have naming inconsistencies.  We can simply rename these 3 Servers to be suffixed with Service instead to become consistent.

Regarding terminology, I view Services as components participating in a larger system.  The Services provide protocol functionality.  I view a server as a standalone daemon process which can expose 1 or more services.  I don't like the use of the term Server except in conjunction if need be with the ApacheDS process which is not just an LDAP server but can expose all these services (the big system).

The [ApacheDS] class is just the LDAP server, containing two services : Ldap and Ldaps, instances of the [LdapService] class, plus a reference to the underlying DirectoryService.

Over the past couple of years this is what we've reduced [ApacheDS] class to but in the past it was the envelope for all the services including LDAP, LDAPS, NTP, Kerberos, Changepw, DNS, and DHCP (never integrated).

I think we could have a better scheme. Here are some of the conceptual problems I have :

1) The [LdapService] class has nothing to do with the [DirectoryBackedService], which shares some information we don't need for a Ldap server.

Perhaps it's shifted away but technically it is directory backed as well.  Question is what does it mean to be directory backed? Is it backed by the DirectoryService or by the LdapService.  If we look at the code a [DirectoryBackedService] is one which has access to the [DirectoryService].  This allows the [LdapService] to store information in the [DirectoryService] which is essentially the entry store while exposing the LDAP protocol as it's primary objective.

Enrique had come up with various parameters that were to be backed in the DIT for two purposes:

  (1) for service component configuration
  (2) for service component runtime data storage
The runtime data storage is important for services like DNS which stores the DNS records in the DIT.  The configuration obviously will store the configuration for the component.  Question is: Does [DirectoryBackedService] map to #1, #2, or both.  I think Enrique was wise to think of #1 as not relavent to this categorization.  I think he intended #2 to be the primary distinction which classified a service as a [DirectoryBackedService].

This makes sense at yet another level.  These components may not use raw LDAP for configuration access.  They may use some configuration interface which is a facade over LDAP based access to the config data.  API's like commons config, the OSGi config interfaces, or even a natural configuration bean model specific to the service driven by LDAP ORM can be used. Regardless, ultimately these interfaces will access and store their information in the DIT.

2) The [NtpServer] is not depending on a DirectoryService, but extends the [AbstractProtocolService] class, which holds the DirectoryService instance

This is a good example of a service that is #1 but not #2. The service will store configuration data in the DIT but does not maintain runtime data it needs to perform it's primary function in the DIT.  Again though access to the DIT through the [DirectoryService] will be need to perform configuration functions.

3) The ApacheDS class is the LDAP server, relying on two instances of the [LdapService] class, one for LDAP, the other one of LDAPS, plus has a reference to the DirectoryService, which is already present in the [AbstractProtocolService] instance

IMO, ApacheDS is a server that should encapsulate all the services including others unrelated to LDAP like Kerbose for example. The intention was to have a server that performed much like MS' Active Directory except services like Kerberos and DNS were in process services instead of external processes. 

4) Last, not least, the ApacheDS class is still in a apacheds-server-jndi project when we don't have any more JNDI in the picture

Right.  The thing will all this is we need a high level architectural vision and need to work towards that instead of evolving as we go.  The problem is in reaching consensus on this.  We should draw this out.  Discuss it and solidify it so we can look at this vision to check that we are always heading in the right direction with respect to features and design decisions as well as the consistency of names of classes and components.

How can we fix all those concerns ?

I think the best approach is to have multiple levels of architectural diagrams depicting our grand vision for what we're building here are Directory.  Once we do this and agree on it then we can use this to guide our decissions.  At this point we have inconsistent fluid models that are mutating over time.  We have no solid footing or perspective. We need this perspective otherwise we will be building more inconsistences as time progresses.  These problems will eventually make it insurmountable to autocorrect with refactoring eventually leading to the need for a full rewrite.

Item #4 should be easy to fix : rename the project to apacheds-server-ldap instead of apacheds-server-jndi

These are small tactical changes that do not fix the main problem.  We need to repost our overall vision then these tactical changes and their correctness can be measured.

For item #3, I would suggest to get rid of the DirectoryService reference, as it's already present in the LdapService instances.

Maybe you're right but why react to this we can fix all this when we have more clarity.

I would add an intermediate class between the [AbstractProtocolService] class and the [DirectoryBackedService] class :

(ProtocolService) o--- [AbstractProtocolService] <-- [AbstractDirectoryBackedService] <-- [DirectoryBackedService] <-- [KdcServer]
(ProtocolService) o--- [AbstractProtocolService] <-- [AbstractDirectoryBackedService] <-- [DirectoryBackedService] <-- [DnsServer]
(ProtocolService) o--- [AbstractProtocolService] <-- [AbstractDirectoryBackedService] <-- [DirectoryBackedService] <-- [ChangePasswordServer]
(ProtocolService) o--- [AbstractProtocolService] <-- [AbstractDirectoryBackedService] <-- [LdapService]
(ProtocolService) o--- [AbstractProtocolService] <-- [NtpServer]

Regarding item #1, I would also removed the DirectoryService reference, as it's already present into the [AbstractProtocolService]  instance.

Does it makes sense to you ?

Nothing makes sense until we draw out the vision and make sure we all agree and align.  We could be going down a path that makes it more difficult to return from.  Regarding your comments though I think you make very good points.  I just don't know if you are right until we agree on a few things which should not take forever to do.

First let's answer these questions and see if we can agree:

(1) What is the main goal of the apacheds subproject?

(2) Is it strictly concerned with producing an LDAP service available as a component and packaged as a standalone server?

Having been around for a long time I've always seen ApacheDS as a highly configurable composite server: not just an LDAP[S] server.  This has governed several decisions I've made over years in the design of the server. I think others including yourself have had this view but it might have changed I'm guessing after this email.

So, to me, our ultimate goal with ApacheDS is to have it become a platform for protocol services which back their protocol content in the DIT.  Sometimes there might be exceptions with supporting protocols like NTP which will be needed for Kerberos operation.  Of course an ntp server outside of ApacheDS can provide this service. However our rational was to just provide this simple service so users will not to have to install external NTP servers in addition to ApacheDS for Kerberos to function.  So the exception here is OK but at the end of the day the NTP service still backs it's configuration in the DIT but this does not make it a [DirectoryBackedService].  Understanding this may give some clarity on how we want to model things.

So at a bare minimum ApacheDS must provide LDAP entry storage which corresponds to this [DirectoryService]. It is not a protocol service however and perhaps chosing this name for it was a mistake which leads to confusion.

Let's discuss this stuff in more detail and get a good picture out that we can use to govern our decisions so they are more correct in leading us to our goal.