directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <elecha...@gmail.com>
Subject Re: JDBM + MVCC LRUCache concern, take 2
Date Wed, 25 Apr 2012 23:45:00 GMT
Le 4/5/12 1:09 AM, Emmanuel Lécharny a écrit :
> Le 4/5/12 12:43 AM, Selcuk AYA a écrit :
>> On Wed, Apr 4, 2012 at 3:22 PM, Emmanuel 
>> Lécharny<elecharny@gmail.com>  wrote:
>>> It's systematic, and I guess that the fact we now pond the RdnIndex 
>>> table
>>> way more often than before (just because we don't call anymore the
>>> OneLevelIndex) cause the cache to get filled and not released fast 
>>> enough.
>> do we hold a cursor open while this code gets stuck? I would think we
>> hold a cursor open and moduify quite a bit of jdbm btree pages for
>> this kind of behavior to happen.
>
> I'll check that.
>>> As we don't set any size for the cache, its default size is 1024. 
>>> For some
>>> of the tests, this mightnot be enough, as we load a lot of entries
>>> (typically the schema elements) plus many others that get added and 
>>> removed
>>> while running tests in revert mode.
>>>
>>> If I increase the default size to 65536, the tests are passing.
>>>
>>> Ok, now, I have to admit I haven't - yet - looked at the LRUCache 
>>> code, and
>>> my analysis is just based on what I saw by quickly looking at the 
>>> code, the
>>> stack traces I have added and some few blind guesses.
>>> However, I think we have a serious issue here. As far as I can tel, 
>>> the code
>>> itself is probably not responsible for this behaviour, but the way 
>>> we use it
>>> is.
>>>
>>> Did I missed something ? Is there anything we can do - except 
>>> increase the
>>> cache size - to get the tests passing fine ?
>>>
>>> I'm more concern about what could occur in real life, when some 
>>> users will
>>> load the server up to a point it just stop responding...
>>   to aovid this issue, we can let the writers allocate more cache
>> pages(rather than keeping the cache size fixed) so that they do not
>> loop waiting for a replaceable cache. However, I would again suggest
>> making sure we do not forget the cursor open. If we forget a cursor
>> open and keep allocating new cache pages for writes, we will have
>> other problems.
> Yeah, I can see how it may affect the tests. I'll definitively 
> investigate this first, before going any further in another direction.
>
> ATM, I'm using a not committed version of JDBM were the default cache 
> size has been changed.
>
> Thanks a lot Selcuk !

So I still have the LRUCache size issue, after having removed the 
SubLevel index. Once I increased the size to 1 << 16, tests are passing.

The failing tests are the SearchAuthorizationIT class' tests.

What happens is that when I add an entry, I update many elements in the 
RdnIndex, as I have to modify the nbDescendant in all its parents. As 
those tests are injecting a lot of entries, so they do a lot of 
modifications in the RdnIndex.

I checked that all the cursors are correctly closed.

Any clue ?

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com


Mime
View raw message