On Wed, May 23, 2012 at 1:36 AM, Emmanuel Lécharny <elecharny@gmail.com> wrote:
Le 5/23/12 12:01 AM, Selcuk AYA a écrit :

I  do not know the OSGI jargon but I believe,  at the end, changing
these should reduce to something like this:

1) quiesce the necessary ldap operations: The simplest thing to do
would be to block all operations quiesce all the outstanding
operations. In some cases, it would be enough to block only searches.
There might be some cases where you do not need any blocking at all. I
do not

I'm not comfortable with the idea of quiscing the ldap operations. The server is supposed to work 24x7, and if we do add some way to modify the backend config on the fly, then the user should not notice that.
Quiscing the Ldap Operations is necessary at some point. Either in DefaultOperationManager level of JdbmPartition level. If we handle that other than DefaultOperationManager level, then we and 3th party developers houldhave to ensure locking is ensured in consistent manner. I believe quiscing the operations at DefaultOperationManager is our best shot, and reconfiguration is not something occurs frequently. Clients will only get late answers to their query when some reconfiguration is in place. Not a big deal IMO? 

See my comments below.

2) do the configuration: In the end I am assuming the configuration is
delegated to the component itself to do the reconfig.

3) unblock the operations.

For index add:

* queisce any ldap operation that might modify the data. Then notify
the partition to add the index. Partition will scan the master table
and build the index and swap its indexed attributes. Then unlock the
Adding an index should be done while the server is still able to process requests. The thing is that the newly added server should not be available until it has been created. The real problem is that we won't be able to parse the master table fast enough to have all the subscequent modifications into it, so this is a two steps operation :
- do a full scan search on the master table, and create the additional index. We should be safe here, because we will use the current version of the master table when we start the search.
- then for every modification done after the beginning of step 1, apply those changes to the index. If we have some more changes since we started this step, then do it again.
What happens if some modification is done while we're executing step2 ! Again, we need locking at some level.

Now, we can use this index.

For index delete:
* block all operations, and remove the index from the indexes list.
Maybe commit this change in partition config. unlock changes. Then
continue deleting  index file.

Here, it's simpler. The operation using this index will continue to use it, until they are done. New operations won't be allowed to use it. Once the number of requests using this index is 0, then the index can be deleted. I don't think we need to block any operation here.

If we enter the deletion code while some operation is using indexes without waiting for that operation to exit, this'll be clearly a concurrency problem. Operation might reference to a deleted index at some point. We need to wait for all operations to return, we shouldn't allow any new operation using this index until we deleted the index. So we can set some locking around index retrieval code, however this would be hard to implement sacrifice to prevent contention. We can't run away from contention at Jdbm, we can only divide it to sublevels anyway.

For suffixdn change:

* I believe this is a rename operation. again block all operations and
do the rename. Then unblock.
changing teh suffix DN is just a matter of modifying the configuration, and the DnNode structure. The problem is that we rebuild all the entries' DN using the partition suffixDn, so we have to be careful when doing so. The request being processed when the rename occurs should be fulfilled with the initial suffixDn.
I have a question here:
Does the suffix dn should be compatible with context entry for that partition ?

Working directory change;
* Block all opreations and let the parition change its config dir. For
jdbm, this would be a copy of it working directory.

Here, this is an administrative task. Not sure we want to do that on a running server...
Yeah we shouldn't move partition files around directories. I believe if partition path is changed, it must trigger partition implementation to look at specified directory to initialize itself from scratch based on given directory's content.

I recommend looking at the things we discussed with Alex on this thread. We should implement some immutability concept for some component types. According to this discussion here is what we may do for JdbmPartition.

* Any reconfiguration will try to create a new instance of JdbmPartition with new configuration.
* If instance creation is failed due to wrong configuration, reconfiguration is aborted
  We will switch runtime instance with created instance while quiscing Operations, and initialize the DirectoryService and new Partition reference.

That way:
* we won't change code of JdbmPartition at all.
* We'll let operations run while we're creating new instance, removing contention at some degree.
* Contention will only occur while we switch runtime references of Partition and initialize new reference inside DirectoryService.

But to do that, we must be sure that side-by-side references can be created with same configuration. For example for LdapServer whis wouldn't be possible, because of its 'network socket' usage, it wouldn't be possible to create new instance if it for reconfiguration operation.

Emmanuel Lécharny