lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Closing IndexWriter can be very slow on large indexes
Date Mon, 01 Aug 2011 12:37:59 GMT
On Mon, Aug 1, 2011 at 8:04 AM, Simon Willnauer
<simon.willnauer@googlemail.com> wrote:
> On Mon, Aug 1, 2011 at 12:57 AM, kiwi clive <kiwi_clive@yahoo.com> wrote:
>> Hi Mike,
>>
>> The problem was due to close().  A shutdown was calling close() which seems to cause
lucene to perform a merge. For a busy very large index (with lots of deletes and updates),
the merge process could take a very long time to complete (hours). Calling close(false) solved
the problem as this appears to close the index without performing the merge. At least that
is my understanding of things !
>>
>
> passing false to IW#close(boolean) will prevent the close call to
> block on merges. If there are background merges in flight those merges
> will be performed nevertheless. While this will not corrupt your index
> you will have dead files lurking around in your index directory if you
> shutdown you app and background threads are killed.

Actually, any running background merges are in fact aborted, and their
files should be cleaned up, before IW.close(false) returns.  IW will
not wait for the merges unless you pass true (which is the default if
you call IW.close()).

> Basically, if you call close IW will flush its internal ram buffer to
> disk creating one new segment (Lucene 3.x) and possibly multiple new
> segments (lucene 4.0). This flush process can take up some time too
> plus this flush can trigger a new merge too. Passing false to
> IW#close(boolean) will also prevent the IW from kicking off a new
> merge due to the flushed segment(s).

Right, passing false also prevents new merges from starting up (as
well as aborting any running ones).

However: you should be careful to sometimes call IW.close(true), else
you have a starvation problem because IW will never have a chance to
complete the big merges, and net/net you are wasting resources because
IW will launch the merges and keep aborting them (if you always pass
false to IW.close).

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message