lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <>
Subject Re: Poor memory performance over a large index
Date Fri, 17 Jun 2005 00:05:08 GMT
: Strange, but opening a second IndexSearcher and running another query
: consumes another 560M of RAM.
: In our case the results are always sorted by some column(s).
: The app is supposed to be a multithreaded multiuser environment.
: At the begining the design was that each user session has its own
: IndexSearcher.
: But later we've made an observation that the first time an IndexSearcher
: is sorting on specific column, it takes significantly more time then the
: next sorts on the same column.
: This forms a performance penalty when changing the sort column. That's
: why we've decided to keep a distinct opened IndexSearcher for each
: possible sort column.
: And each session upon executing its query will obtain the "right"
: IndexSearcher that is quick on sorting by that specific column.

wow.  this seems way more complicated then it needs to be.

the reason you see the memory footprint go up so much with each search
that uses a sort on a field, is because of the FieldCache.  You shoudl
search arround for an explanation/description of what/why it is and how it
works, but the bottomline is that time/memory cost is only paid once per
field you search on *per* IndexReader ... so opening a seperate
IndexSearcher/IndexReader per user is definitely not hte best way to go,
but open a seperate IndexSearcher per sort field should also be
unneccessary.  you should be able to use *one* IndexSearcher for all
searches, with the same memory footprint, and reduced complexity.

: The problem is that (having in mind the abovementioned 325M & 540M) the
: memory gets exhausted too quickly which really leaves no room for
: multiple users running queries simultaneously.

the only advice i have is to increase your heap size, and minimize the
number of IndexSearchers.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message