lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: OutOfMemory ParallelMultisearcher
Date Wed, 17 Mar 2010 11:20:37 GMT
Hi


Caching searchers at some level should help keep memory usage down -
and will help performance too.  Searchers themselves don't generally
consume large amounts of memory, but if you've got loads of them then
obviously things will add up.

Unless you can change the whole design of your app (single index with
a user field that you use as a query filter to restrict users to
"their" data?) you may be stuck with giving the app more memory or
restricting the number of concurrent searchers.


---
Ian.


On Tue, Mar 16, 2010 at 10:00 PM, Jamie <jamie@stimulussoft.com> wrote:
> Hi There
>
> I have an index which is 36 GB large. When I perform  eight simultaneous
> searches (performed by JMeter) on the index, an OutOfMemory error occurs.
> Since I need to potentially search across multiple indexes and those indexes
> can change from one search query to the next, each user has their own
> ParallelMultiSearcher object. Before each search operation, I reconstruct
> the ParrallelMultisearcher with the appropriate Searchers to each of the
> indexes that need to be included for that particular search query.
>
> The problem is that requiring each user to have their own
> ParallelMultisearcher seems to limit the number of  users that can use the
> system at the same time.
>
> While experimenting, when I make the ParallelMultiSearcher static, the same
> object used by all users, the OutOfMemory problem goes away and I am able to
> execute 50 simultaneous searches. The problem I have is I cannot make
> ParallelMultisearcher static, since the specific indexes used are variable
> from one search query to the next. I initially thought one could just cache
> the underlying Searchers and all would be okay, but this does not appear to
> be the case.
>
> My question: Will ParallelMultisearcher tend to consume a large amount of
> memory by itself when used on large indices? If so, do you have any
> suggestions on how I might support the above scenario (i.e. when the indexes
> used change from one query to the next)
>
> Thanks in advance
>
> Jamie
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message