lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Benge" <todd.be...@gmail.com>
Subject Re: OutOfMemory Problems Lucene 2.4 / Tomcat
Date Wed, 05 Nov 2008 00:45:40 GMT
Thanks Pablo.

I'll be flying to New Orleans tomorrow for ApacheCon and would love
the opportunity to talk with others about architectures others are
using.

Todd



On 11/4/08, PabloS <pablo.a.saavedra@gmail.com> wrote:
>
> Sure Todd,
>
> the idea basically consist in the following:
>
> - Subclassing FIeldSortedHitQueue and calling support with an empty
> SortField array: this disables caching because the comparators are retrieved
> during construction
> - Creating a new SortComparatorSource that creates the sort comparators
> similarly to FieldCacheImpl but without performing any caching
> - Creating a new HitCollector that behaves just like TopFieldDocCollector
> but uses my custom FieldSortedHitQueue. The tricky part was creating a
> TopFieldDoc instance (which has default visibility constructor. I managed to
> do it creating an empty TopFieldDocCollector, asking the TopFieldDocs and
> modifiying that instance.
> - Overriding the Searcher's search(Weight, Filter, int, Sort) so it uses my
> HitCollector (I use the Hits class, which internally calls that method)
>
> That way, you completely override field caching and you'll suffer the
> performance consequences. What I did was making the SortComparatorSource a
> spring bean and adding a caching interceptor to it. You an plug your
> favorite cache library in, in my case ehcache.
>
> I hope you find this useful.
> Regards.
>
>
> tbenge wrote:
>>
>> Pablo,
>>
>> Would you mind adding a little more detail about how you're working
>> around the problem?
>>
>> I'm still evaluating our different options so am interested in what you
>> did.
>>
>> Todd
>>
>> On Mon, Nov 3, 2008 at 2:37 PM, PabloS <pablo.a.saavedra@gmail.com> wrote:
>>>
>>> Thanks hossman, but I've already 'solved' the problem without the need to
>>> patch lucene. I had to code a bit around Lucene's visibility restrictions
>>> but I've managed to completely skip the field caching mechanism and add
>>> ehcache to it.
>>>
>>> At the moment it seems to be working quite well, although not as fast as
>>> it
>>> was when lucene performed the caching.
>>>
>>> Thanks a lot for the info anyway.
>>> Regards.
>>>
>>> hossman wrote:
>>>>
>>>>
>>>> : I'm having a similar problem with my application, although we are
>>>> using
>>>> : lucene 2.3.2. The problem we have is that we are required to sort on
>>>> most of
>>>> : the fields (20 at least). Is there any way of changing the cache being
>>>> used?
>>>>
>>>> there is a patch in Jira that takes a completley different approach
>>>> towards dealing with sorting by using the *stored* values of the
>>>> documents
>>>> that match...
>>>>
>>>> https://issues.apache.org/jira/browse/LUCENE-769
>>>>
>>>> ...in theory it's "better" for use cases where:
>>>>
>>>>   1) you are confident you are going to get back a "small" result set in
>>>>      proportion to the total size of your index.
>>>>   2) you need to support sorting on "lots" of fields and don't have
>>>> enough
>>>>      ram for all of the FieldCaches of those fields
>>>>
>>>> So far the only person to test it out (and post comments) is the patch
>>>> submitter, but if other people report success it might be worth adding
>>>> as
>>>> a contrib.
>>>>
>>>> (Disclaimer: it looks like the patch was updated after my last
>>>> review/comments.  i have not read, nor claim any opinion about the
>>>> currently attached patch)
>>>>
>>>>
>>>>
>>>>
>>>> -Hoss
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20311386.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/OutOfMemory-Problems-Lucene-2.4---Tomcat-tp20236834p20326783.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

-- 
Sent from Gmail for mobile | mobile.google.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message