Hi Emmanuel,

On Mon, Oct 31, 2011 at 8:52 PM, Emmanuel Lecharny <elecharny@gmail.com> wrote:
On 10/31/11 10:17 AM, Alex Karasulu wrote:
On Sun, Oct 30, 2011 at 7:02 PM, Göktürk Gezer<gokturk.gezer@gmail.com>wrote:

Hi Devs,

I'd like to do some brainstorming about Interceptor extension mechanism
which we're about to implement. Ideas those will come out from that thread
will enlighten our way on other extension points of the ApacheDS.

The first main issue is whether we're going to preserve old standalone
ApacheDS or not?

Let's be really careful about the common terminology we will be using for
this discussion. When you say "standalone ApacheDS", I automatically think
about the standalone ApacheDS maven module that holds the ApacheDS main()
application launcher.

In a previous thread Pierre recommended keeping such a module but modifing
it so that it starts up an OSGi container which launches the ApacheDS
bundles. However the user experience does not change much. This merely
allows users to start up ApacheDS from the command line as before but
behind the scenes an OSGi container is launched as the main() application.
This is one option and I think I like this approach because this way we
don't have to maintain a monolithic application startup and a OSGi startup.

Preserving API in nonOSGI environments had purpose. And with the very
simple method we've used in API, we can also preserve ApacheDS's nonOSGI
version too.

Sure we can do anything but will it add additional complexity and overhead
for maintaining code and what's the gain that we get? Will it make
debugging easier to maintain this non-OSGi configuration?

Doing it will have pros and cons.

Ahh there you go with the pros and cons :-).

The main advantage will be of course letting existing user semantics

It appears to me that you're talking about more than just the application
launcher verses the OSGi based launch sequence. I guess you're referring to
all the configuration mechanism changes that will impact the different

To me these usage semantic changes are small compared to the overall gains
we will get.

But implementing extendibility without breaking ApaheDS's nonOSGI version
will put some limits on our extension capability on individual components.
Besides those limitations, we'll also end up having two different runtime
behaviour of ApacheDS. Documentation will be obviously painful. And some
aspects won't be implementable if we choose to preserve nonOSGI version.
For example, OSGI can give us the opportunity to don't restart server after
some changes on configuration, which will be perfect improvement of course.
But to make this we must go fully OSGI. List is long.

Yeah I prefer to forget about maintaining the non-OSGi version. The
"standalone ApacheDS" maven module can launch the OSGi container or Karaf
then have the OSGi environment fire up ApacheDS as a full OSGi application.

We've talked about it before but i wanted to ask that question again,
because its an alive topic and subject to change. Maybe some of yours mind
could have been changed, and i want to know if it did. If we proceed like
what we've been talked earlier, then I'm going to make ApacheDS (not
shared) an OSGI application, and use all of its loveliness.


To talk about Interceptors specifically, here is the design that i'm
thinking. To manage OSGI heavy dynamic nature, we must implement our
Interceptor related code in the closest module that is using Interceptors,
which is InterceptorChain in our case. First step will be implement a
component called InterceptorHub which is responsible for keeping list of
all Interceptors installed in the container(means installed in OSGI as
bundle). This hub will create an instance of every Interceptor per every
ApacheDS instance and keep it.

When an InterceptorChain want to iterate through interceptors, it will get
them from InterceptorHub rather then DirectoryService.getInterceptors(). So
it will get the most recent list of interceptors every time.


Just a question: why not just make the InterceptorChain act as or perform
the duties of the hub? I guess the hub is intended as an internal component
of the chain that hides some details relating to OSGi from the chain?

Currently in our config.ldif file, there is an interceptors entries. We
can leave those entries there for specifying mandatory interceptors. So if
any of them is not exists at the time server is starting we can throw an
exception saying necessary interceptor is not exist. But we won't be have
to list additional interceptors in that list. They will be automatically
attached to the active ApacheDS instances.

We'll also change the Interceptor implementations to be an IPojo
component. Those components will publish two main properties. First is
their ordering and the second is their bypass list.

Ordering is something we've talked about but i couldn't get a clear
explanation about it. What i suggest is like this: we publish three kind of
ordering from interceptor: strict, level and relax. "strict" means strict
ordering, this will be published from core interceptors and that strict
order will create a level. If we publish it as level we must also attach an
index with it, such as level-3. If we publish it as relax it means it can
be called anywhere in the chain after the strict ones. So lets suppose we
documented the core interceptor ordering like:

1- InterceptorA
2- InterceptorB
3- InterceptorC

So there is 3 level in interceptor chain. If somebody implement an
additional interceptor named InterceptorD and publish its ordering
as level-2, it is saying it must be called after InterceptorA(level-1). So
the new list will be:

1- InterceptorA
2- InterceptorD
3- InterceptorC

I didnt't numbered B as 3 but rather i grouped it with the InterceptorD,
putting InterceptorD above it. Why? What happens if vendor wants to
implement two custom interceptor named InterceptorD and InterceptorE and
want 2 of them to be called right after InterceptorA(after 1th level) but
want InterceptorD to be called first. So we must suborder those 2
interceptors. That's why i treated the first core list as levels rather
than orders ! So the ordering which is published from custom interceptor is
not sufficient to describe good ordering. We must also publish subordering
which specifies the ordering between that specific vendor's interceptors.

This is rather interesting. This approach never occurred to me but it's
appealing. I'll give this some thought and also would like to hear from
others on the list about it.

Jumping on Göktürk idea, some proposal.

We all know that the core interceptors are to be executed in a given order. Calling the NormalizationInterceptor after the CollectiveAttributeInterceptor does not make any sense.

OTOH, we allow the admin to modify the order, simply by changing the configuration, and it could lead to some bad bad things...

We wont let admin to change order for core ones.

What if we create an implicit order by defining which interceptor is allowed before each interceptor ? For instance, this is the default order :

1  normalizationInterceptor
2  authenticationInterceptor
3  referralInterceptor
4  aciAuthorizationInterceptor
5  defaultAuthorizationInterceptor
6  exceptionInterceptor
7  operationalAttributeInterceptor
8  keyDerivationInterceptor
9  passwordHashingInterceptor
10 schemaInterceptor
11 collectiveAttributeInterceptor
12 subentryInterceptor
13 eventInterceptor
14 triggerInterceptor

We could obtain the same order by replacing the ads-interceptororder AT by a ads-previousInterceptor :

1  normalizationInterceptor ( previous : null )
2  authenticationInterceptor ( previous : normalizationInterceptor )
3  referralInterceptor ( previous : authenticationInterceptor )
4  aciAuthorizationInterceptor ( previous : referralInterceptor )
5  defaultAuthorizationInterceptor ( previous : aciAuthorizationInterceptor, referralInterceptor ) (here, the aciAuthorizationInterceptor may be disabled)
6  exceptionInterceptor ( previous : defaultAuthorizationInterceptor )
7  operationalAttributeInterceptor ( previous : exceptionInterceptor )
8  keyDerivationInterceptor ( previous : operationalAttributeInterceptor )
9  passwordHashingInterceptor ( previous : keyDerivationInterceptor )
10 schemaInterceptor ( previous : passwordHashingInterceptor )
11 collectiveAttributeInterceptor ( previous : schemaInterceptor )
12 subentryInterceptor ( previous : collectiveAttributeInterceptor )
13 eventInterceptor ( previous : subentryInterceptor )
14 triggerInterceptor ( previous : eventInterceptor )

Now, for interceptors that has no specific requirement, we can set the previous AT to the previous AT it will follow.

The key is that we must distinguished between critical inteceptors (the level-1 interceptors) and others. We enforce the position only within one level (ie, level-1 interceptors orders are checked, then level-2, then level-3). So we can add a non critical interceptor (ie level-2 or 3) in the chain between two critical interceptors, without creating any trouble in the critical interceptor order : the check for level-1 interceptors' order is done ignoring level-2 interceptors.

thoughts ?

At first i thought that linked buckets approach. This is easier to implement and it is intuitive. By this way, 3th parties can also make their one of custom interceptor to be dependent on the some other custom interceptor. So that, they can create the same ordering resolution as with the method i proposed.

But there are some problems with that approach:

* First of all we're creating an implicit dependency here. Every interceptor in the current codebase is not mandatory. But some other custom interceptor implementation may refer to those which are not core. If this "noncore but apacheds-made" interceptor is missing in runtime. The ordering is impossible. We can't even warn the user or throw exception. Because in OSGI, everything can come and go at anytime. When we can't find the ads-previousInterceptor at install time of particular interceptor, we can't raise an error, we have to wait for it.But if that dependency never comes, then we can't sort the interceptor list. Serious problem!

* Secondly, somebody may want to write custom version of the core interceptors. But with using linked dependencies with names, this is impossible without changing the source code of the interceptors those are dependent on the replaced one. That will create very big weakness on our extendibility.

* And i believe, possessing abstraction of level into interceptor ordering is good for users and custom implementors. Because that way we can say for example, "at level1, X service is maintained", "at level2, Y service is maintained." and so on... So that, they inject their custom interceptor to the right level of their choice without knowing the name of the interceptor that implemented that logic.
So again, this issues are the concern of  "How much extendibility we'll offer?" question.

So lets suppose InterceptorD(ordering="strict-2",subordering="foo1"),
InterceptorE(ordering="strict-2", subordering="foo2"). Both interceptors
want to be executed after 1th level(means after InterceptorA). But they
also publish subordering with the same token, "foo". So they will be called
after 1.level which is InterceptorA, but also they will be called in order
InterceptorD - interceptorE with the help of their subordering string.

We can also depend on string ordering of Interceptor class names here, but
i think its not good. We can also merge ordering and subordering in one,
like (ordering="level2-foo1"). This way, we can define infinite subordering
by separating them with dashes.

So this is my way to give interceptor writers a way to describe their
interceptor's order in the chain. If i'm missing something here please say
it and i'll put it in consideration too.

Very interesting ideas. I think we're in total agreement regarding going
fully OSGi and making the standalone launcher just start the container then

I think I really like this ordering scheme you've proposed but I have to
give it some time and thought. Also interested in other commentary.  This
looks great and thanks for your effort.

Emmanuel Lécharny