lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephane James Vaucher <vauch...@cirano.qc.ca>
Subject Re: Swapping Indexes?
Date Tue, 17 Aug 2004 17:17:10 GMT
On Tue, 17 Aug 2004, Patrick Burleson wrote:

> Forward back to list.
> 
> 
> ---------- Forwarded message ----------
> From: Patrick Burleson <pburleson@gmail.com>
> Date: Tue, 17 Aug 2004 11:30:19 -0400
> Subject: Re: Swapping Indexes?
> To: Stephane James Vaucher <vauchers@cirano.qc.ca>
> 
> Stephane,
> 
> Thank you for the ideas. I'm going about implenting idea 1 (I like the
> idea of leaving the temp index around for recovery), but I have a
> question reguarding your original index. Do you just copy over the
> temp index and don't worry abou cleaning up the old index directory?

Actually, I use a IndexWriter in overwrite mode on the master dir and 
merge the temp dir. This cleans up the old master.
 
> Right now I have my code deleting the files in the main index
> directory after telling the search controller to switch to the temp
> index. But by doing that, I need to manage existing searches and not
> break them while they are running. I also still run into the open
> files problem on Windows when trying to delete a file one of the
> searchers has open before it's closed.

I used to way some time (~1 minute) for all searches on the old master to 
finish after redirecting to the temp dir, then I would switch to the new 
master. 

> Thoughts?

If you apply a lease-like contract with your searchers where they 
borrow a reference to a searcher and then hand it back to the manager, 
you can probably trace your open files.

HTH,
sv
 
> Patrick
> 
> 
> 
> 
> On Mon, 16 Aug 2004 18:22:20 -0400 (EDT), Stephane James Vaucher
> <vauchers@cirano.qc.ca> wrote:
> > I've tried two options that seem to work:
> >
> > 1) Have a singleton that is responsible that will control your searchers.
> > This controller can temporarilly redirect your searchers to
> > c:/temp/myindex, allowing you to copy you index to c:/myindex. After that
> > process completes, your controller can tell your searchers to use
> > c:/myindex, allowing you to then erase your temp index.
> >
> > If you index nightly, you can always *not* erase your tmp dir, your index
> > process will do this automatically if you create your IndexWriter with
> > the overwrite option. This way, you can have a backup index if there is
> > a system failure at some point (like when you copy/move directories).
> >
> > 2) Use an incremental index. Regularly, I scan my files, see if there are
> > modification/additions and update my master index. Removing from the
> > master index, adding to a temp dir, then merging. I haven't seen any
> > weirdness on windows with this process.
> >
> > HTH,
> > sv
> >
> >
> >
> > On Mon, 16 Aug 2004, Patrick Burleson wrote:
> >
> > > I've read in the docs about updating an index and its suggestion
> > > reguarding swapping out indexes with a directory rename.
> > >
> > > Here's my question, how to do this when searches are running live?
> > >
> > > Say I have a directory that holds the current valid index:
> > >
> > > C:\myindex
> > >
> > > and when I'm running my nightly process to generate the index, it gets
> > > temporarily indexed to:
> > >
> > > C:\temp\myindex
> > >
> > > How can I very quickly replace C:\myindex with C:\temp\myindex?
> > >
> > > I can't simply do a rename since C:\myindex will likely have open
> > > files. (Gotta love windows)
> > >
> > > And I can't delete all files in myindex, again because of the open files issue.
> > >
> > > Any ideas?
> > >
> > > Thanks,
> > > Patrick
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> > >
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message