lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <>
Subject [jira] Commented: (LUCENE-1703) Add a waitForMerges() method to IndexWriter
Date Fri, 19 Jun 2009 22:14:07 GMT


Shai Erera commented on LUCENE-1703:

one thread may call addDocument() (or maybeMerge() to be more to the point)
this thread could result in the SerialMergeScheduler to start merging (addDocument() won't
return until this merge completes)

Again, if autoCommit=false, how can this happen? I thought that if autoCommit=false, no commit
happens and therefore no segment merging?

bq. I think it makes sense to expose waitForMerges in IW (instead of duplicating the code
in every merge scheduler). We may be able to then deprecate CMS.sync?

I guess I'm still not convinced what simplicity would that bring. For Tim's use case, just
two threads, using SMS, that might work. But for the general use case, or one which uses multiple
indexing threads, one of which may call commit() at some point, another daemon may run optimize(),
I dunno, this would still require syncing all threads around that waitForMerges call, if the
intent is to prevent document additions while merges occur. Therefore this method is not expected
to make my life any easier, except that if I need to report "no merges are running" or make
a decision based on "no merges are running" I should have one thread call this waitForMerges()

Of course, still playing the devil's advocate, you could call waitForMerges() which will return
immediately b/c there are no merges to do or that are running, and soon as that happens, a
context switch also happens, and an innocent addDocument will trigger a 50-segments merge,
at which point whatever you thought to do b/c there are no merges, will hit the exact scenario
you were trying to avoid all that time :).

I'll admit though that it's late here (1 AM), and perhaps I'm not seeing this clearly. And
.. I still don't understand how if autocommit=false, and addDocument/deleteDocument can trigger
a merge.

> Add a waitForMerges() method to IndexWriter
> -------------------------------------------
>                 Key: LUCENE-1703
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Tim Smith
>         Attachments:,
> It would be very useful to have a waitForMerges() method on the IndexWriter.
> Right now, the only way i can see to achieve this is to call IndexWriter.close()
> ideally, there would be a method on the IndexWriter to wait for merges without actually
closing the index.
> This would make it so that background merges (or optimize) can be waited for without
closing the IndexWriter, and then reopening a new IndexWriter
> the close() reopen IndexWriter method can be problematic if the close() fails as the
write lock won't be released
> this could then result in the following sequence:
> * close() - fails
> * force unlock the write lock (per close() documentation)
> * new IndexWriter() (acquires write lock)
> * finalize() on old IndexWriter releases the write lock
> * Index is now not locked, and another IndexWriter pointing to the same directory could
be opened
> If you don't force unlock the write lock, opening a new IndexWriter will fail until garbage
collection calls finalize() the old IndexWriter
> If the waitForMerges() method is available, i would likely never need to close() the
IndexWriter until right before the process being shutdown, so this issue would not occur (worst
case scenario, the waitForMerges() fails)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message