directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@gmail.com>
Subject Re: [Studio] Using JDBM for secondary cache
Date Wed, 02 Apr 2008 21:32:25 GMT
Stefan Seelmann wrote:
> Hi Alex,
>
>
> Alex Karasulu schrieb:
>> Stefan S.,
>>
>> Do we use a secondary cache for Studio? Just wondering because of the 
>> performance issues someone noted on the user list when dealing with a 
>> very large directory.  
>
> no, we don't use a cache for Studio. I tried to use ehcache long time 
> ago but it was really slow when it swaps entries to the disk and back.
Whatever cache you use, as soon as you hit the disk, performances will 
be awfull. The question is : do we accept those terrible performance, or 
should we accept an OOM ? Now, with JDBM, you have a middle term 
situation. If you use an appropriate size for the JDBM cache, it will 
only contain the tree, not the entries, leading to a great perf 
improvement. You will still have to hit the disk to get the data, but 
you will avoid a lit of disk access.

Of course, it depends on the kind of operation you want to run. We 
discussed a lot with Pierre-Arnaud about the idea to integrate ADS 
_into_ studio, gaining a direct benefit of being able to play faked 
operations locally, and also to use its internal cache.

To be discussed, of course...
>
> Today there we have the following:
> - a HashMap<String, IEntry>: with the DN as key and the entry as value 
> (IEntry is a Studio internal interface with some implementations)
> - a HashMap<IEntry, AttributeInfo>: as soon as the attributes of an 
> entry were loaded an AttributeInfo containing all attributes and other 
> information is created and put to this map
> - a hashMap<IEntry, ChildrenInfo>: as soon as child entries are loaded 
> a ChildrenInfo containing all child entries is created and put to this 
> map.
>
> For sure, this does scale well. With the default VM parameters (64MB 
> heap) you could load about 30,000 entries to get an OutOfMemory :-(((

What about WeakHashap ?
>
> I also think that the switch from the old DN/RDN implementation to the 
> shared-ldap LdapDN/Rdn implementation costs some memory. We should 
> consider to do some test for performance and memeory consumtion.
Performance is ok, but we may eat more memory with LdapDN, as we keep 
two forms of the DN (user provided and normalized). In your case, I 
would say it's unoticable, when compared with the other attributes 
(especially big binary values).

If we were to dig, I would not pick this place...
>
>
>> The idea occurred to me that the JDBM code for JdbmTable and 
>> JdbmIndex could potentially be used by Studio to help solve some of 
>> the caching problems.  If this is something you think will help we 
>> can move this code into shared so both the server and Studio can 
>> leverage them.
>
> Yeah, that sounds great. You want me to look how the JDBM code works? 
> I guess we will have some time at ApacheCon for that.
Yes, sure ! This is what AC is for !


-- 
--
cordialement, regards,
Emmanuel L├ęcharny
www.iktek.com
directory.apache.org



Mime
View raw message