lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Earwin Burrfoot" <ear...@gmail.com>
Subject Re: [jira] Commented: (LUCENE-855) MemoryCachedRangeFilter to boost performance of Range queries
Date Thu, 04 Dec 2008 22:21:49 GMT
It would be cool to be able to explicitly list subreaders that were
added/removed as a result of reopen(), or have some kind of
notification mechanism.
We have filter caches, custom field/sort caches here and they are all
reader-bound. Currently warm-up delay is negated by reopening and
warming up in background, before switching to the new reader/caches,
but it still limits our minimum between-reopens delay.

On Fri, Dec 5, 2008 at 01:03, robert engels <rengels@ix.netcom.com> wrote:
> The biggest benefit I see of using the field cache to do filter caching, is
> that the same cache can be used for sorting - thereby improving the
> performance and memory usage.
>
> The downside I see is that if you have a common filter that is built from
> many fields, you are going to use a lot more memory, as every field used
> needs to be cached. With my code you would only have a single "bitset" for
> the filter.
>
> On Dec 4, 2008, at 4:00 PM, robert engels wrote:
>
>> Lucene-831 is far more comprehensive.
>>
>> I also think that by exposing access to the sub-readers it can be far
>> simpler (closer to what I have provided).
>>
>> In the mean-time, you should be able to use the provided class with a few
>> modifications.
>>
>> The "reload the entire cache" was a deal breaker for us, so I came up the
>> attached. Works very well.
>>
>> On Dec 4, 2008, at 3:54 PM, Uwe Schindler wrote:
>>
>>> I am looking all the time to LUCENE-831, which is a new version of
>>> FieldCache that is compatible with IndexReader.reopen() and invalidates
>>> only
>>> reloaded segments. In each release of Lucene I am very unhappy, because
>>> it
>>> is still not in. The same problem like yours is if you have a one million
>>> documents index that is updated by adding a few documents each half hour.
>>> If
>>> you use sorting by a field, whenever the index is reopened and you really
>>> only a very small segment is added, nevertheless the complete FieldCache
>>> is
>>> rebuild, very bad :(.
>>>
>>>
>>> So I think the ultimative fix would be to hopefully apply LUCENE-831 soon
>>> and also use LUCENE-1461 as RangeFilter cache.
>>>
>>> -----
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>> ________________________________________
>>> From: robert engels [mailto:rengels@ix.netcom.com]
>>> Sent: Thursday, December 04, 2008 9:39 PM
>>> To: java-dev@lucene.apache.org
>>> Subject: Re: [jira] Commented: (LUCENE-855) MemoryCachedRangeFilter to
>>> boost
>>> performance of Range queries
>>>
>>> I can't seem to post to Jira, so I am attaching here...
>>>
>>> I attached QueryFilter.java.
>>>
>>> In reading this patch, and other similar ones, the problem seems to be
>>> that
>>> if the index is modified, the cache is invalidated, causing a complete
>>> reload of the cache. Do I have this correct?
>>>
>>> The attached patch works really well in a highly interactive environment,
>>> as
>>> the cache is only invalidated at the segment level.
>>>
>>> The MyMultiReader is a subclass that allows access to the underlying
>>> SegmentReaders.
>>>
>>> The patch cannot be applied, but I think the implementation works far
>>> better
>>> in many cases - it is also far less memory intensive. Scanning the bitset
>>> could also be optimized very easily using internal skip values.
>>>
>>> Maybe this is completely off-base, but the solution has worked very well
>>> for
>>> us. Maybe this is a completely different issue and separate incident
>>> should
>>> be opened ?
>>>
>>> is there any interest in this?
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>



-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
Mime
View raw message