I don't know the internal of a Partition but can't you implement some kind of lazy loading, ie an entry is loaded when requested by an LDAP request into the AvlPartition ? Btw, this should be used with some cache mecanism.
atm, the LdifPartition is using an internal AvlPartition to manage the entries and index in memory. That means we have to load all the entries from disk, and store them in memory. Of course, when some modifications are done, they are stored on disk too.
The problem with this approach is that it limits the number of entries we are likely to handle in such a partition to a few thousands. It's fine for the schema partition and the configuration partition, which must be in memory anyway for performance reasons. But in the near future, it's a real limitation.
There is also a technical issue with the LdifPartition API : we can't switch the current AvlPartition to use another one. It would be really cool if we could inject a wrapped partition into the LdifPartition, replacing the default AvlPartition, but atm, it's not possible, as we don't have an interface for that. We can't use Partition, it's too generic, and we can't use BTreePartition, as it does not expose the store.
What would be cool is to improve this part of the Partition API. It's not urgent, but I think it has to be done before 2.0. It's not a big change too.