directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Göktürk Gezer <>
Subject Re: Problem of JDBM being hard(impossible) to reconfigure once it is initialized
Date Wed, 23 May 2012 15:01:31 GMT
On Wed, May 23, 2012 at 1:36 AM, Emmanuel Lécharny <>wrote:

> Le 5/23/12 12:01 AM, Selcuk AYA a écrit :
>  I  do not know the OSGI jargon but I believe,  at the end, changing
>> these should reduce to something like this:
>> 1) quiesce the necessary ldap operations: The simplest thing to do
>> would be to block all operations quiesce all the outstanding
>> operations. In some cases, it would be enough to block only searches.
>> There might be some cases where you do not need any blocking at all. I
>> do not
> I'm not comfortable with the idea of quiscing the ldap operations. The
> server is supposed to work 24x7, and if we do add some way to modify the
> backend config on the fly, then the user should not notice that.
Quiscing the Ldap Operations is necessary at some point. Either in
DefaultOperationManager level of JdbmPartition level. If we handle that
other than DefaultOperationManager level, then we and 3th party developers
houldhave to ensure locking is ensured in consistent manner. I believe
quiscing the operations at DefaultOperationManager is our best shot, and
reconfiguration is not something occurs frequently. Clients will only get
late answers to their query when some reconfiguration is in place. Not a
big deal IMO?

> See my comments below.
>> 2) do the configuration: In the end I am assuming the configuration is
>> delegated to the component itself to do the reconfig.
>> 3) unblock the operations.
>> For index add:
>> * queisce any ldap operation that might modify the data. Then notify
>> the partition to add the index. Partition will scan the master table
>> and build the index and swap its indexed attributes. Then unlock the
>> operations.
> Adding an index should be done while the server is still able to process
> requests. The thing is that the newly added server should not be available
> until it has been created. The real problem is that we won't be able to
> parse the master table fast enough to have all the subscequent
> modifications into it, so this is a two steps operation :
> - do a full scan search on the master table, and create the additional
> index. We should be safe here, because we will use the current version of
> the master table when we start the search.
> - then for every modification done after the beginning of step 1, apply
> those changes to the index. If we have some more changes since we started
> this step, then do it again.
What happens if some modification is done while we're executing step2 !
Again, we need locking at some level.

> Now, we can use this index.
>> For index delete:
>> * block all operations, and remove the index from the indexes list.
>> Maybe commit this change in partition config. unlock changes. Then
>> continue deleting  index file.
> Here, it's simpler. The operation using this index will continue to use
> it, until they are done. New operations won't be allowed to use it. Once
> the number of requests using this index is 0, then the index can be
> deleted. I don't think we need to block any operation here.

If we enter the deletion code while some operation is using indexes without
waiting for that operation to exit, this'll be clearly a concurrency
problem. Operation might reference to a deleted index at some point. We
need to wait for all operations to return, we shouldn't allow any new
operation using this index until we deleted the index. So we can set some
locking around index retrieval code, however this would be hard to
implement sacrifice to prevent contention. We can't run away from
contention at Jdbm, we can only divide it to sublevels anyway.

>> For suffixdn change:
>> * I believe this is a rename operation. again block all operations and
>> do the rename. Then unblock.
> changing teh suffix DN is just a matter of modifying the configuration,
> and the DnNode structure. The problem is that we rebuild all the entries'
> DN using the partition suffixDn, so we have to be careful when doing so.
> The request being processed when the rename occurs should be fulfilled with
> the initial suffixDn.

I have a question here:
Does the suffix dn should be compatible with context entry for that
partition ?

>> Working directory change;
>> * Block all opreations and let the parition change its config dir. For
>> jdbm, this would be a copy of it working directory.
> Here, this is an administrative task. Not sure we want to do that on a
> running server...

Yeah we shouldn't move partition files around directories. I believe if
partition path is changed, it must trigger partition implementation to look
at specified directory to initialize itself from scratch based on given
directory's content.

I recommend looking at the things we discussed with Alex on this thread. We
should implement some immutability concept for some component types.
According to this discussion here is what we may do for JdbmPartition.

* Any reconfiguration will try to create a new instance of JdbmPartition
with new configuration.
* If instance creation is failed due to wrong configuration,
reconfiguration is aborted
  We will switch runtime instance with created instance while quiscing
Operations, and initialize the DirectoryService and new Partition reference.

That way:
* we won't change code of JdbmPartition at all.
* We'll let operations run while we're creating new instance, removing
contention at some degree.
* Contention will only occur while we switch runtime references of
Partition and initialize new reference inside DirectoryService.

But to do that, we must be sure that side-by-side references can be created
with same configuration. For example for LdapServer whis wouldn't be
possible, because of its 'network socket' usage, it wouldn't be possible to
create new instance if it for reconfiguration operation.

> --
> Regards,
> Cordialement,
> Emmanuel Lécharny

View raw message