lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1703) Add a waitForMerges() method to IndexWriter
Date Fri, 19 Jun 2009 20:34:07 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721987#action_12721987
] 

Shai Erera commented on LUCENE-1703:
------------------------------------

I'm just playing the devil's advocate here. 

I'm still not sure that this method is needed. A MergeScheduler is given to IndexWriter, and
therefore your code should be able to sync() on it, whether it's ConcurrentMergeScheduler
or a different one which does background merges, since you control its creation. If it's something
that needs to improve on CMS, let's improve it (like documenting sync()).

bq. My primary use case for this is to stabilize an index prior to performing a backup operation

Still .. if you call any of the methods I've mentioned, you can change your code to call the
doWait variants, except for commit(). If commit() is what you're after, wouldn't it be better
to add a commit(doWait) so that you can call it w/ doWait=true?

BTW, Mike McCandless has written a great article on Hot Backups w/ Lucene, which you can find
here: http://www.manning.com/free/green_HotBackupsLucene.html (need to register to Manning
first). If you're not familiar w/ it, I suggest you read it since it shows a nice and elegant
ways to perform backups, w/o the need to sync your writer.

I'm planning to try it out. One thing you can do, after calling snapshot() is to compare the
files returned from snapshot() to the ones in the backup and: (1) delete the ones in backup
that do not appear in the returned list of files, and copy over the files that do not exist
in the backup (except for .lock). The article is really good and self explanatory.

> Add a waitForMerges() method to IndexWriter
> -------------------------------------------
>
>                 Key: LUCENE-1703
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1703
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Tim Smith
>         Attachments: IndexWriter.java.diff, IndexWriter.java.diff
>
>
> It would be very useful to have a waitForMerges() method on the IndexWriter.
> Right now, the only way i can see to achieve this is to call IndexWriter.close()
> ideally, there would be a method on the IndexWriter to wait for merges without actually
closing the index.
> This would make it so that background merges (or optimize) can be waited for without
closing the IndexWriter, and then reopening a new IndexWriter
> the close() reopen IndexWriter method can be problematic if the close() fails as the
write lock won't be released
> this could then result in the following sequence:
> * close() - fails
> * force unlock the write lock (per close() documentation)
> * new IndexWriter() (acquires write lock)
> * finalize() on old IndexWriter releases the write lock
> * Index is now not locked, and another IndexWriter pointing to the same directory could
be opened
> If you don't force unlock the write lock, opening a new IndexWriter will fail until garbage
collection calls finalize() the old IndexWriter
> If the waitForMerges() method is available, i would likely never need to close() the
IndexWriter until right before the process being shutdown, so this issue would not occur (worst
case scenario, the waitForMerges() fails)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message