lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From PabloS <pablo.a.saave...@gmail.com>
Subject Re: OutOfMemory Problems Lucene 2.4 / Tomcat
Date Tue, 04 Nov 2008 17:14:36 GMT

Sure Todd,

the idea basically consist in the following:

- Subclassing FIeldSortedHitQueue and calling support with an empty
SortField array: this disables caching because the comparators are retrieved
during construction
- Creating a new SortComparatorSource that creates the sort comparators
similarly to FieldCacheImpl but without performing any caching
- Creating a new HitCollector that behaves just like TopFieldDocCollector
but uses my custom FieldSortedHitQueue. The tricky part was creating a
TopFieldDoc instance (which has default visibility constructor. I managed to
do it creating an empty TopFieldDocCollector, asking the TopFieldDocs and
modifiying that instance.
- Overriding the Searcher's search(Weight, Filter, int, Sort) so it uses my
HitCollector (I use the Hits class, which internally calls that method)

That way, you completely override field caching and you'll suffer the
performance consequences. What I did was making the SortComparatorSource a
spring bean and adding a caching interceptor to it. You an plug your
favorite cache library in, in my case ehcache.

I hope you find this useful.
Regards.


tbenge wrote:
> 
> Pablo,
> 
> Would you mind adding a little more detail about how you're working
> around the problem?
> 
> I'm still evaluating our different options so am interested in what you
> did.
> 
> Todd
> 
> On Mon, Nov 3, 2008 at 2:37 PM, PabloS <pablo.a.saavedra@gmail.com> wrote:
>>
>> Thanks hossman, but I've already 'solved' the problem without the need to
>> patch lucene. I had to code a bit around Lucene's visibility restrictions
>> but I've managed to completely skip the field caching mechanism and add
>> ehcache to it.
>>
>> At the moment it seems to be working quite well, although not as fast as
>> it
>> was when lucene performed the caching.
>>
>> Thanks a lot for the info anyway.
>> Regards.
>>
>> hossman wrote:
>>>
>>>
>>> : I'm having a similar problem with my application, although we are
>>> using
>>> : lucene 2.3.2. The problem we have is that we are required to sort on
>>> most of
>>> : the fields (20 at least). Is there any way of changing the cache being
>>> used?
>>>
>>> there is a patch in Jira that takes a completley different approach
>>> towards dealing with sorting by using the *stored* values of the
>>> documents
>>> that match...
>>>
>>> https://issues.apache.org/jira/browse/LUCENE-769
>>>
>>> ...in theory it's "better" for use cases where:
>>>
>>>   1) you are confident you are going to get back a "small" result set in
>>>      proportion to the total size of your index.
>>>   2) you need to support sorting on "lots" of fields and don't have
>>> enough
>>>      ram for all of the FieldCaches of those fields
>>>
>>> So far the only person to test it out (and post comments) is the patch
>>> submitter, but if other people report success it might be worth adding
>>> as
>>> a contrib.
>>>
>>> (Disclaimer: it looks like the patch was updated after my last
>>> review/comments.  i have not read, nor claim any opinion about the
>>> currently attached patch)
>>>
>>>
>>>
>>>
>>> -Hoss
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20326783.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message