lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Klaas" <>
Subject Re: index file size threshold affecting search performance?
Date Thu, 29 Mar 2007 01:50:56 GMT
On 3/28/07, Scott Oshima <> wrote:
> So I assumed a linear decay of performance as an index got bigger.
> For some reason when going from an index size of 1.89 to 1.95 gigs
> dramatically increased cpu across all of our servers.
> I was thinking of splitting the 1.95 index into 2 separate indexes  and
> using a multisearcher on those parts?

PS, you might find this helpful:

You should tell us what you have done and what was the unexpected
consequences (and possibly your hypothesis as to why).  Instead,
you've only told us your hypothesis, but not:

 - whether the increase is due to more documents or more data per document
 - whether the increase is in the indexed content or store field content
 - what format of additional data is being stored and what you are doing with it
 - where the performance degradation is occurring (query or document retrieval)
 - if the former, what type of queries are being used
 - if the latter, how many documents are being retrieved

I could see raw index size having an effect if the active set _just_
fits in the OS buffer cache, and you've pushed it over the edge.  But
in that case, I would expect the performance degradation to manifest
as increased io.

One guess is that you added a compressed field, which can be cpu intensive.

Guessing is painful for us and for you.  Provide more details! :)


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message