lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Bamford <chris.bamf...@talktalk.net>
Subject Closing IndexWriter can be very slow on large indexes
Date Tue, 26 Jul 2011 15:33:48 GMT
Hi

I think I must be doing something wrong, but not sure what.  

I have some long running indexing code which sometimes needs to be shutdown in a hurry.  To
achieve this, I set a shutdown flag which causes it to break from the loop and call first
abort() and then close().  The problem is that with a large index (say, 15Gb) in Lucene 2.3.2,
it can take over an hour.  (Yes, I know I should be on a later version of Lucene, but that's
another issue - we are stuck with this for now!).

The IW is opened in autoCommit mode and mergeFactor=10.

During this closedown stage, the indexes are being constantly updated by Lucene itself, making
me suspect it could be merging.

Firstly, can someone explain what it is doing under the covers that takes so long? (And any
action I can take to get around it)

Second, if I were to rebuild the code with say, Lucene 3 and run it in compatibility mode
with the 2.3.2 indexes, would I have a richer set of tools I could use to overcome the issue?

Thanks,

- Chris

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message