lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Klaas" <mike.kl...@gmail.com>
Subject Re: index file size threshold affecting search performance?
Date Thu, 29 Mar 2007 01:50:56 GMT
On 3/28/07, Scott Oshima <soshima@gmail.com> wrote:
> So I assumed a linear decay of performance as an index got bigger.
>
> For some reason when going from an index size of 1.89 to 1.95 gigs
> dramatically increased cpu across all of our servers.
>
> I was thinking of splitting the 1.95 index into 2 separate indexes  and
> using a multisearcher on those parts?

PS, you might find this helpful:
http://www.catb.org/~esr/faqs/smart-questions.html

You should tell us what you have done and what was the unexpected
consequences (and possibly your hypothesis as to why).  Instead,
you've only told us your hypothesis, but not:

 - whether the increase is due to more documents or more data per document
 - whether the increase is in the indexed content or store field content
 - what format of additional data is being stored and what you are doing with it
 - where the performance degradation is occurring (query or document retrieval)
 - if the former, what type of queries are being used
 - if the latter, how many documents are being retrieved

I could see raw index size having an effect if the active set _just_
fits in the OS buffer cache, and you've pushed it over the edge.  But
in that case, I would expect the performance degradation to manifest
as increased io.

One guess is that you added a compressed field, which can be cpu intensive.

Guessing is painful for us and for you.  Provide more details! :)

-Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message