lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <>
Subject Re: OutOfMemory ParallelMultisearcher
Date Thu, 18 Mar 2010 06:00:50 GMT
Hi Ian

Thanks for the info. Its difficult to reuse searchers as my users are 
performing realtime searches, so I need to open an IndexReader for every 
live search query.

I've since tracked the OutOfMemory issue down to sort on date. I am 
using too high a precision (down to the second) which is causing the 
number of search terms to escalate. Also, I haven't since moved the date 
field in the index over to using Numerics instead of a String value for 
fear of breaking compatibility with my old index format.

I dont yet quite understand the implications of the precisionStep and 
what it all means. If I change my date string to a Numeric integer in 
the format yyyyMMddHHmm, what should the precisionStep value be?


On 2010/03/17 01:20 PM, Ian Lea wrote:
> Hi
> Caching searchers at some level should help keep memory usage down -
> and will help performance too.  Searchers themselves don't generally
> consume large amounts of memory, but if you've got loads of them then
> obviously things will add up.
> Unless you can change the whole design of your app (single index with
> a user field that you use as a query filter to restrict users to
> "their" data?) you may be stuck with giving the app more memory or
> restricting the number of concurrent searchers.
> ---
> Ian.
> On Tue, Mar 16, 2010 at 10:00 PM, Jamie<>  wrote:
>> Hi There
>> I have an index which is 36 GB large. When I perform  eight simultaneous
>> searches (performed by JMeter) on the index, an OutOfMemory error occurs.
>> Since I need to potentially search across multiple indexes and those indexes
>> can change from one search query to the next, each user has their own
>> ParallelMultiSearcher object. Before each search operation, I reconstruct
>> the ParrallelMultisearcher with the appropriate Searchers to each of the
>> indexes that need to be included for that particular search query.
>> The problem is that requiring each user to have their own
>> ParallelMultisearcher seems to limit the number of  users that can use the
>> system at the same time.
>> While experimenting, when I make the ParallelMultiSearcher static, the same
>> object used by all users, the OutOfMemory problem goes away and I am able to
>> execute 50 simultaneous searches. The problem I have is I cannot make
>> ParallelMultisearcher static, since the specific indexes used are variable
>> from one search query to the next. I initially thought one could just cache
>> the underlying Searchers and all would be okay, but this does not appear to
>> be the case.
>> My question: Will ParallelMultisearcher tend to consume a large amount of
>> memory by itself when used on large indices? If so, do you have any
>> suggestions on how I might support the above scenario (i.e. when the indexes
>> used change from one query to the next)
>> Thanks in advance
>> Jamie
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message