lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Giles <>
Subject Re: Concurrency
Date Fri, 20 Feb 2004 17:05:55 GMT
It would be great if we could come up with way to integrate the Lucene 
locking information with something more incremental like rsync.  At Furl ( ) we have this problem in spades because we have 
thousands (and thousands) of indexes that need to be backed up.  Currently, 
we run rsync frequently (i.e. hourly) with "safe" (i.e. stopped server) 
daily snapshots (that rollover, etc.).  Essentially we are playing the odds 
on the frequent backups since indexes are only open for the moment that a 
new item is saved (i.e. they are reopened/closed each time) and we also 
have all of the data to recreate them if need be.  But this is definitely a 
topic we have discussed and it would be nice to have a solution to it.

Any other ideas out there (i.e. a way to check the locks and then retry a 
bit later)?


At 11:43 AM 2/20/2004, you wrote:
>Alan Smith wrote:
>>1. What happens if i make a backup (copy) of an index while documents are 
>>being added? Can it cause problems, and if so is there a way to safely do this?
>This is not in general safe.  A copy may not be a usable index.  The 
>segments file points to the current set of files.  An IndexWriter 
>periodically rewrites the segments file, and then may delete files which 
>are no longer used.  If you copy the segments file, then, before you copy 
>all the files, the segments file is re-written and some files are deleted, 
>your copied index will be incoherent.  Or, vice versa, you might copy the 
>segments file last, and it may refer to newly created files which you 
>failed to copy.
>The safest way to do this is to use an IndexReader to make your backup, 
>with something like:
>   IndexReader reader ="index");
>   IndexWriter writer = new IndexWriter("backup", analyzer, true);
>   writer.addIndexes(new IndexReader[] { reader });
>   writer.close();
>   reader.close();
>This will use Lucene's locking code to make sure all is safe.
>>2. When I create a new IndexSearcher, what method does Lucene use to take 
>>a 'snapshot' of the index (because if i add documents after the search 
>>object is created they dont appear in the search results)?
>It keeps the existing set of files open.  As described above, Lucene 
>modifies an index by adding new files and removing old ones.  But an open 
>index never changes its set of files.
>To unsubscribe, e-mail:
>For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message