directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <>
Subject Re: JDBM + MVCC LRUCache concern, take 2
Date Thu, 26 Apr 2012 09:21:08 GMT
Hi !

so I'm to get used with the fact that cursor must be *carfully* closed! 
I fucked up here, as I forgot one single close after having modifed 
mocally the code. Fixing this magically solved the issue.

Sorry for the noise !

Le 4/26/12 2:08 AM, Selcuk AYA a écrit :
> On Wed, Apr 25, 2012 at 4:45 PM, Emmanuel Lécharny<>  wrote:
>> Le 4/5/12 1:09 AM, Emmanuel Lécharny a écrit :
>>> Le 4/5/12 12:43 AM, Selcuk AYA a écrit :
>>>> On Wed, Apr 4, 2012 at 3:22 PM, Emmanuel Lécharny<>
>>>>   wrote:
>>>>> It's systematic, and I guess that the fact we now pond the RdnIndex
>>>>> table
>>>>> way more often than before (just because we don't call anymore the
>>>>> OneLevelIndex) cause the cache to get filled and not released fast
>>>>> enough.
>>>> do we hold a cursor open while this code gets stuck? I would think we
>>>> hold a cursor open and moduify quite a bit of jdbm btree pages for
>>>> this kind of behavior to happen.
>>> I'll check that.
>>>>> As we don't set any size for the cache, its default size is 1024. For
>>>>> some
>>>>> of the tests, this mightnot be enough, as we load a lot of entries
>>>>> (typically the schema elements) plus many others that get added and
>>>>> removed
>>>>> while running tests in revert mode.
>>>>> If I increase the default size to 65536, the tests are passing.
>>>>> Ok, now, I have to admit I haven't - yet - looked at the LRUCache code,
>>>>> and
>>>>> my analysis is just based on what I saw by quickly looking at the code,
>>>>> the
>>>>> stack traces I have added and some few blind guesses.
>>>>> However, I think we have a serious issue here. As far as I can tel, the
>>>>> code
>>>>> itself is probably not responsible for this behaviour, but the way we
>>>>> use it
>>>>> is.
>>>>> Did I missed something ? Is there anything we can do - except increase
>>>>> the
>>>>> cache size - to get the tests passing fine ?
>>>>> I'm more concern about what could occur in real life, when some users
>>>>> will
>>>>> load the server up to a point it just stop responding...
>>>>   to aovid this issue, we can let the writers allocate more cache
>>>> pages(rather than keeping the cache size fixed) so that they do not
>>>> loop waiting for a replaceable cache. However, I would again suggest
>>>> making sure we do not forget the cursor open. If we forget a cursor
>>>> open and keep allocating new cache pages for writes, we will have
>>>> other problems.
>>> Yeah, I can see how it may affect the tests. I'll definitively investigate
>>> this first, before going any further in another direction.
>>> ATM, I'm using a not committed version of JDBM were the default cache size
>>> has been changed.
>>> Thanks a lot Selcuk !
>> So I still have the LRUCache size issue, after having removed the SubLevel
>> index. Once I increased the size to 1<<  16, tests are passing.
>> The failing tests are the SearchAuthorizationIT class' tests.
>> What happens is that when I add an entry, I update many elements in the
>> RdnIndex, as I have to modify the nbDescendant in all its parents. As those
>> tests are injecting a lot of entries, so they do a lot of modifications in
>> the RdnIndex.
>> I checked that all the cursors are correctly closed.
>> Any clue ?
> Can you provide the following details:
> -on which jdbm table are you having the problem(rdn index, main table?)?
> - approximately how many modifications are you doing on this table
> while you are holding a cursor open( even if the cursor is held open
> legally). Knowing this number would help a lot.
> - is the same problem you had before or did closing the cursors in the
> previous case solve your problem?
> this kind of problem can currently occur if a thread is holding a
> cursor on one table(not necessarily illegally) and the "same" thread
> is modifying the "same" table with many add/delete/update operations.
> I am wondering whether we have a use case like this now. If we have,
> then I can change the code to account for it.
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
> thanks
> Selcuk

Emmanuel Lécharny

View raw message