lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@stimulussoft.com>
Subject Re: OutOfMemory ParallelMultisearcher
Date Thu, 18 Mar 2010 06:00:50 GMT
Hi Ian

Thanks for the info. Its difficult to reuse searchers as my users are 
performing realtime searches, so I need to open an IndexReader for every 
live search query.

I've since tracked the OutOfMemory issue down to sort on date. I am 
using too high a precision (down to the second) which is causing the 
number of search terms to escalate. Also, I haven't since moved the date 
field in the index over to using Numerics instead of a String value for 
fear of breaking compatibility with my old index format.

I dont yet quite understand the implications of the precisionStep and 
what it all means. If I change my date string to a Numeric integer in 
the format yyyyMMddHHmm, what should the precisionStep value be?

Jamie



On 2010/03/17 01:20 PM, Ian Lea wrote:
> Hi
>
>
> Caching searchers at some level should help keep memory usage down -
> and will help performance too.  Searchers themselves don't generally
> consume large amounts of memory, but if you've got loads of them then
> obviously things will add up.
>
> Unless you can change the whole design of your app (single index with
> a user field that you use as a query filter to restrict users to
> "their" data?) you may be stuck with giving the app more memory or
> restricting the number of concurrent searchers.
>
>
> ---
> Ian.
>
>
> On Tue, Mar 16, 2010 at 10:00 PM, Jamie<jamie@stimulussoft.com>  wrote:
>    
>> Hi There
>>
>> I have an index which is 36 GB large. When I perform  eight simultaneous
>> searches (performed by JMeter) on the index, an OutOfMemory error occurs.
>> Since I need to potentially search across multiple indexes and those indexes
>> can change from one search query to the next, each user has their own
>> ParallelMultiSearcher object. Before each search operation, I reconstruct
>> the ParrallelMultisearcher with the appropriate Searchers to each of the
>> indexes that need to be included for that particular search query.
>>
>> The problem is that requiring each user to have their own
>> ParallelMultisearcher seems to limit the number of  users that can use the
>> system at the same time.
>>
>> While experimenting, when I make the ParallelMultiSearcher static, the same
>> object used by all users, the OutOfMemory problem goes away and I am able to
>> execute 50 simultaneous searches. The problem I have is I cannot make
>> ParallelMultisearcher static, since the specific indexes used are variable
>> from one search query to the next. I initially thought one could just cache
>> the underlying Searchers and all would be okay, but this does not appear to
>> be the case.
>>
>> My question: Will ParallelMultisearcher tend to consume a large amount of
>> memory by itself when used on large indices? If so, do you have any
>> suggestions on how I might support the above scenario (i.e. when the indexes
>> used change from one query to the next)
>>
>> Thanks in advance
>>
>> Jamie
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>      
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message