I don't know the internal of a Partition but can't you implement some kind of lazy loading, ie an entry is loaded when requested by an LDAP request into the AvlPartition ? Btw, this should be used with some cache mecanism.


On Tue, Sep 8, 2009 at 11:02 PM, Emmanuel Lecharny <elecharny@apache.org> wrote:

atm, the LdifPartition is using an internal AvlPartition to manage the entries and index in memory. That means we have to load all the entries from disk, and store them in memory. Of course, when some modifications are done, they are stored on disk too.

The problem with this approach is that it limits the number of entries we are likely to handle in such a partition to a few thousands. It's fine for the schema partition and the configuration partition, which must be in memory anyway for performance reasons. But in the near future, it's a real limitation.

There is also a technical issue with the LdifPartition API : we can't switch the current AvlPartition to use another one. It would be really cool if we could inject a wrapped partition into the LdifPartition, replacing the default AvlPartition, but atm, it's not possible, as we don't have an interface for that. We can't use Partition, it's too generic, and we can't use BTreePartition, as it does not expose the store.

What would be cool is to improve this part of the Partition API. It's not urgent, but I think it has to be done before 2.0. It's not a big change too.

thoughts ?

cordialement, regards,
Emmanuel Lécharny

La mélancolie c’est communiste
Tout le monde y a droit de temps en temps
La mélancolie n’est pas capitaliste
C’est même gratuit pour les perdants
La mélancolie c’est pacifiste
On ne lui rentre jamais dedans
La mélancolie oh tu sais ça existe
Elle se prend même avec des gants
La mélancolie c’est pour les syndicalistes
Il faut juste sa carte de permanent

Miossec (2006)