Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: pass (athena.apache.org: domain of elecharny@gmail.com
 designates 209.85.218.222 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=CdSeM16VI2Is0QD6cSlrzAXCV+aamEcyRmcXnQYYA5gvluMDfbzQxTThmH2MuTUzZv
         nTC6hAejbpYq+57fL1gpgh27xREoe5JN9gWFpJJqDOR1IMnrkMijM2jmnHXmSXs1TYNJ
         SWWLtJKdvu92joX0VXMPidXtLFFNWAr+V0uzQ=
Sender: Emmanuel Lecharny <elecharny@gmail.com>
Message-ID: <4B0D0B79.8080206@nextury.com>
Date: Wed, 25 Nov 2009 11:48:25 +0100
From: Emmanuel Lecharny <elecharny@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: Apache Directory Developers List <dev@directory.apache.org>
Subject: Re: [ApacheDS] [Schema] Update mechanism
References: <a32f6b020911231320r47852d3ak436f3a024b6177d5@mail.gmail.com>
In-Reply-To: <a32f6b020911231320r47852d3ak436f3a024b6177d5@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Alex Karasulu wrote:
> Thinking more about how to update the schema and the mechanics involved
> along with the various trade offs.  Here's how this should progress:
>
> Schema Change Steps
>
> (1) A schema change operation arrives and changes are performed on a cloned
> set of registries where the impact of the change is calculated in terms of
> the total number of indirect dependents affected.
>   
When you talk about 'indirect dependents affected', you mean affected 
SchemaObjects, I guess.
> (2a) If the operation is consistent then a schema-txn-begin message is sent
> to all listeners informing them of the OIDs impacted by the schema change
> calculated in the first step.  This lets objects dependent on these schema
> entities to respond to the situation. This message is delivered
> synchronously.  Access to the old Registries is provided before the swap in
> case the object needs them via the SchemaManager handle if they are exposed.
>   
There is an issue here, if the message is synchronous, as many of the 
ServerEntries (or constants used all over the code) may be in memory, 
without any thread using them. Some of the use cases are :
- objects in cache (we have many different cache, like the referral 
cache, the alias cache, etc)
- objects currently being built, but not completed, waiting for some 
more data from the client side (in the codec)
- objects which has already been checked against the old schema, but are 
not yet processed by the backend, and may become invalid

There is also another potential problem if the modification impacts the 
index (like if we removed an indexed AttributeType).

Anyway, sending synchronous messages is simply impossible.
> (2b) If the operation leaves the schema in an inconsistent state then no
> message is sent to schema listeners and the schema change is rejected.
>   
That I agree on.
> (3) The schema registries are swapped out.
> (4) The SchemaManager then sends a schema-txn-commit message to all
> listeners to now be able to access the modified schema elements they're
> interested in from the newly swapped in registries.  This message is
> delivered synchronously.
>   
See point 2a above.
>
> Schema Change Responses
>
> (1) Invalidate the object in question so all using/run processes and caches
> can reset or update themselves via the acquisition of a new valid object to
> continue the process.
>   
Clearing the cache is an option, but for aliases or referals, it will 
request a new search to get back all the aliases or referrals. Can be 
costly... But maybe the problem is not that this operation is costly, 
but that it's the way we handle aliases and referrals which is wrong...
> (2) Make the object depending on schema objects valid by updating stale
> references and hope for the best.  Sometimes timing will cause certain
> failures but this is really OK with us since this is a rare situation to
> update the schema.
>   
Assuming that an synchronous message being sent to the objects can't be 
done, the other option would be to send this synchronous message to the 
session and to the cache managers, as they have direct access to the 
objects. Let's say the session is not active (no thread to process it), 
and let's say it's reactivated at some point : the first thing to do 
would be to check if the schema has changed, and if so, propagate the 
new schema to all the related objects. That mean we must register 
aother  listener in the objects to react to this session modification. 
the very same for the cache managers.

About the cache, we have to use a better strategy than the one we are 
using? It's insane to have so many caches all over the server. We should 
use one generic cache system where we store all the different elements 
we want to cache, and have a dedicated thread to manage the global 
cache. But the first step would be to list all the caches we are using...

Interesting times !

-- 
--
cordialement, regards,
Emmanuel L�charny
www.iktek.com
directory.apache.org