jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ard Schrijvers <a.schrijv...@onehippo.com>
Subject Re: Searching for a property
Date Mon, 11 Jan 2010 12:19:28 GMT
On Mon, Jan 11, 2010 at 1:07 PM, Dennis van der Laan
<d.g.van.der.laan@rug.nl> wrote:
> Hello Ard,
>
> We rewrote a part of our virtual path handling, and now store both the
> virtual path itself, and the lower-case equivalent (we really need the
> not-lowercased path). All queries are now done on the lowercased virtual
> path and indeed (!) everything stays fast, even after a million virtual
> paths. We'll try to keep away from the lower-case function and similar
> functions.

as long as it is a single term lookup in Lucene, it is always fast,
almost regardless the number of terms there are

>
> Thanks very much for all your help!

You're welcome,

Ard

>
> Dennis
>
> Ard Schrijvers wrote:
>> On Thu, Dec 17, 2009 at 10:59 PM, Dennis van der Laan
>> <d.g.van.der.laan@rug.nl> wrote:
>>
>>> Dennis van der Laan wrote:
>>>
>>
>>
>>> See the increase of time spent on the execution: 400+ ms instead of 7ms.
>>> And this is not a single incident, I see this increase on all queries
>>> like the above.
>>>
>>> The memory of the JVM should not be a problem, it's set to 2Gb and only
>>> 800Mb is used at the moment the queries are slow. Restarting the
>>> application does not help either.
>>>
>>
>> No, this seems logical to me. The memory is consumed by internal
>> lucene term enums. I am quite sure what your issue is, but did not
>> test it, nor ever tried it myself. But, I have always wondered *how*
>> the fn:lower-case could have been implemented efficiently in
>> Jackrabbit. It doesn't fit into my understanding of how inverted
>> indexes work, what Lucene is in the end. So, I am happy that my
>> understanding was correct, and unhappy that fn:lower-case does (again,
>> from top of my head and looking at code only) not scale to well.
>>
>> I think in your setup a lot of time is spend in the CaseTermQuery,
>> which traverses all your 1 million virtualpaths first and lowercase
>> it. This cannot scale (nor in cpu, nor in memory).
>>
>> So, would you like to give me an indication about the query execution
>> time without the fn:lower-case? I think it will drop to < 1 ms.
>>
>> I think you should try to get away without using the fn:local-name if
>> this works for you. Just make sure that you store the virtualpath
>> property always as lower-case: then, you are fine
>>
>>
>>> Again, any help will be appreciated.
>>>
>>
>> let me know if this helped,
>>
>> Regards Ard
>>
>>
>>> Dennis
>>>
>>>
>>>>> Furthermore, of course, index size matters as well
>>>>>
>>>>>
>
>
> --
> Dennis van der Laan
>
>

Mime
View raw message