lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Kristensson <mark.kristens...@smartsheet.com>
Subject IndexWriter.close() performance issue
Date Mon, 01 Nov 2010 22:55:45 GMT
Hello,

One of our  Lucene indexes has started misbehaving on indexWriter.close and I'm searching
for ideas about what may have happened and how to fix it.

Here's our scenario:
- We have seven Lucene indexes that contain different sets of data from a web application
are indexed for searching by end users
- A java service runs twice a minute to pull changes from SQL DB queue tables and update the
relevant Lucene index(es)
- The two largest indexes (3.4GB and 3.8GB in size with 8 million and 6 million records, respectively)
contain similar sets of data, but are structured differently for different consumption (e.g.
one has an All field for general purpose searching, the other does not; one has numeric fields
for ranges, the other does not, etc.)
- We expunge deletes from our indexes twice per day
- A  couple of weeks ago, one of the two large indexes became very slow to close after each
round of changes is applied by our indexing service. Specifically, all of our indexes usually
close in no more than 200 milliseconds, but this one index is now taking 6 to 8 seconds to
close with every single pass (and is leading to delays which are affecting end users). 

Questions from my attempts to troubleshoot the problem:
- Has anyone else seen behavior similar to this? What did you do to resolve it?
- Does the size of an index or its documents (record count, disk size, avg document size,
max document size, etc.) have any correlation to the length of time it takes to close an index?
- We are not currently optimizing any of our indexes on a regular basis, could that have any
impact upon indexing operations (e.g. indexWriter.close())? My understanding is that optimization
only affects search performance, not indexing performance and to date we have not seen any
need to optimize based upon the performance of the search queries.

Thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message