lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Smith (JIRA)" <>
Subject [jira] Commented: (LUCENE-1703) Add a waitForMerges() method to IndexWriter
Date Fri, 19 Jun 2009 20:54:07 GMT


Tim Smith commented on LUCENE-1703:

MergeScheduler does not provide a sync() method in its abstract class. Therefore, an implementation
could be used that would not derive from or provide a sync() method

If it did, this would be a more than acceptable solution instead of putting the method on
the IndexWriter. (i potentially want to allow configuration to specify what MergeScheduler
to use, and if sync() isn't specified on the base class, i can't reliably use it (will have
to do instanceof checks, and a new concurrent based scheduler not derived from ConcurrentMergeScheduler
breaks this)

Also, the SerialMergeScheduler may be in use and be running due to another thread calling
addDocument triggering a merge, but i want my second thread to wait for that merge to complete
(this would require implementing sync() on SerialMergeScheduler to allow other threads to
sync on the merge as well)

optimize() is not a viable option to use to wait for merges as this could take a long time
close() requires reopening the IndexWriter (which i am trying to avoid in the first place)
expungeDeletes(true) is doing more work that is actually required (this could be a viable
solution, however its a bit odd to "waitForMerges()" by calling a method called expungeDeletes())

I will try getting access to the article you suggest, however i still expect i will want this
method at the end of the day (and will end up using expungeDeletes(true) if it never gets
added (with copious comments indicating that expungeDeletes() is actually being used to wait
for background merges)

Another use for waitForMerges() is as a more lightweight version of optimize().
this will allow blocking adding more documents until background merges are complete periodically,
preventing from throttling the IO on the indexer from being hit too hard. Could call IndexWriter.maybeMerge()
followed by IndexWriter.waitForMerges()

> Add a waitForMerges() method to IndexWriter
> -------------------------------------------
>                 Key: LUCENE-1703
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Tim Smith
>         Attachments:,
> It would be very useful to have a waitForMerges() method on the IndexWriter.
> Right now, the only way i can see to achieve this is to call IndexWriter.close()
> ideally, there would be a method on the IndexWriter to wait for merges without actually
closing the index.
> This would make it so that background merges (or optimize) can be waited for without
closing the IndexWriter, and then reopening a new IndexWriter
> the close() reopen IndexWriter method can be problematic if the close() fails as the
write lock won't be released
> this could then result in the following sequence:
> * close() - fails
> * force unlock the write lock (per close() documentation)
> * new IndexWriter() (acquires write lock)
> * finalize() on old IndexWriter releases the write lock
> * Index is now not locked, and another IndexWriter pointing to the same directory could
be opened
> If you don't force unlock the write lock, opening a new IndexWriter will fail until garbage
collection calls finalize() the old IndexWriter
> If the waitForMerges() method is available, i would likely never need to close() the
IndexWriter until right before the process being shutdown, so this issue would not occur (worst
case scenario, the waitForMerges() fails)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message