lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanislav Jordanov <ste...@sirma.bg>
Subject Poor memory performance over a large index
Date Thu, 16 Jun 2005 13:19:37 GMT
We are in a similar situatuin.
The index contains about 1,000,000 docs and its total size is 31G (note: 
Gigabytes, not Megabytes).
The problem is not the search speed - it is the memory usage.
Opening the first IndexSearcher and running a query consumes about 325M 
of RAM
Strange, but opening a second IndexSearcher and running another query 
consumes another 560M of RAM.
In our case the results are always sorted by some column(s).
The app is supposed to be a multithreaded multiuser environment.
At the begining the design was that each user session has its own 
IndexSearcher.
But later we've made an observation that the first time an IndexSearcher 
is sorting on specific column, it takes significantly more time then the 
next sorts on the same column.
This forms a performance penalty when changing the sort column. That's 
why we've decided to keep a distinct opened IndexSearcher for each 
possible sort column.
And each session upon executing its query will obtain the "right" 
IndexSearcher that is quick on sorting by that specific column.
The problem is that (having in mind the abovementioned 325M & 540M) the 
memory gets exhausted too quickly which really leaves no room for 
multiple users running queries simultaneously.

Any idea that may help will be appreciated :)

Regards
Stenly

Erik Hatcher wrote:

>
> On Jun 16, 2005, at 4:08 AM, JM Tinghir wrote:
>
>> I have a 25 Mb index and was wondering if it would be better to divide
>> it in about 10 indexes and search in it with MutliSearcher.
>> Would searching be faster this way?
>> The indexing would be faster I guess, as it is getting slower and
>> slower while indexes get bigger.
>> But searching?
>
>
> I think keeping it in one index is preferable in this situation.   
> Perhaps you need to optimize the index?  Could you qualify a bit more  
> about what is slow?  What types of queries?  How "slow" are they?
>
>     Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message