lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Kolstad" <chri...@ovitas.no>
Subject Re: Best practice for updating an index when reindexing is not an option
Date Fri, 11 Jul 2008 07:07:22 GMT
Hi.

First, thanks for the reply.

Why does SubversionUpdate require shutting down the IndexSearcher?  What
> goes wrong?
>

SubversionUpdate requires shutting down the IndexSearcher in our current
implementation because the old index files are deleted in the tag we're
switching to. Sorry, just realised that my last mail didn't state that we
don't in fact to an "svn up", but rather an "svn switch". Thus, when we try
to perform the update SubversionUpdate fails due to file lock issues when
trying to update (deleting the old lucene files) the lucene index directory
(The relevant code for the update action is quoted below).

You might want to switch instead to rsync.
>

I'm hoping I won't have to, firstly because I'm more familiar with
subversion, secondly because that would require me to configure rsync for
windows, and I'm still not sure if that will help anything with the file
lock issues we're trying to avoid.

A Lucene index is fundamentally write once, so, syncing changes over should
> simply be copying over new files and removing now-deleted files.  You won't
> be able to remove files held open by the IndexSearcher, but, once the
> IndexSearcher restarts you'd then be able to delete those files on the next
> sync.


So I should be able to run the switch and then restart the IndexSearcher,
instead of turning off the IndexSearcher, run the switch, turn on the
IndexSearcher. I'd see how that would work with a linux box, having a bit
more trouble seeing how I will get it to work with a windows box (and my
live server is unfortunately a Windows 2003 box), Subversion keeps running
into file lock issues when I switch from one tag to the other if I try to
keep the search active. With Lucene 2.1 it even ran into file lock issues
after I'd disabled the search and was performing the switch. Now, when we're
using the Lucene 2.3.2 jar the lock issues has mostly gone (once in 3
months, instead of every switch/update).

Current code:

disableSearch(request); //Sets the SearchActive boolean to false
>
> Search searcher =
> (Search)ctx.getAttribute(FelleskatalogenStartupServlet.SEARCH);
>       if (searcher != null) {
>         searcher.clear();
>      }

           String latestTag =
> SubversionUtil.getInstance().getLatestTag(getTagUrl(request));
>
>             SubversionUtil.getInstance().runSwitch(getRoot(request),
> getTagUrl(request) + "/" + latestTag);
>
>             if (log.isDebugEnabled()) {
>                 log.debug("Index set to " + getRoot(request) + "/lucene");
>             }
>
>             ctx.setAttribute(FelleskatalogenStartupServlet.SEARCH, new
> Search(getRoot(request) + "/lucene"));
>             ctx.setAttribute(FelleskatalogenStartupServlet.SEARCHACTIVE,
> new Boolean(true));
>


BR,

Christopher

On Thu, Jul 10, 2008 at 4:27 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:



>
> Why does SubversionUpdate require shutting down the IndexSearcher?  What
> goes wrong?
>
> You might want to switch instead to rsync.
>
> A Lucene index is fundamentally write once, so, syncing changes over should
> simply be copying over new files and removing now-deleted files.  You won't
> be able to remove files held open by the IndexSearcher, but, once the
> IndexSearcher restarts you'd then be able to delete those files on the next
> sync.
>
> Mike
>
>
> Christopher Kolstad wrote:
>
>  Hi.
>>
>> Currently using Lucene 2.3.2 in a tomcat webapp. We have an action
>> configured that performs reindexing on our staging server. However, our
>> live
>> server can not reindex since it does not have the necessary dtd files to
>> process the xml.
>>
>> To update the index on the live server we perform a subversion update on
>> the
>> lucene index directory.
>> Unfortunately this makes it necessary to stop the IndexSearcher while the
>> SubversionUpdate is doing its thing.
>>
>> Presently we've had a requirement from our customer to not disable search.
>>
>> So my idea was to copy the index directory to another directory and then
>> switch the IndexSearcher from the original index directory to the
>> temporary
>> directory.
>> Then perform the Subversion update, and when done, switch the
>> IndexSearcher
>> back to the original (now, updated) index directory.
>>
>> Does anyone have any other suggestions on how to update the index
>> directory
>> from subversion without having to disable the IndexSearcher?
>>
>> BR
>> Christopher
>>
>> --
>> Regards,
>> Christopher Kolstad
>> =============================
>> |100 little bugs in the code, debug one, |
>> |recompile, 101 little bugs in the code |
>> =============================
>>
>> E-mail: chriswk@ifi.uio.no (University)
>> christopher.kolstad@gmail.com (Home)
>> chriswk@ovitas.no (Job)
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Regards,
Christopher Kolstad
=============================
|100 little bugs in the code, debug one, |
|recompile, 101 little bugs in the code |
=============================

E-mail: chriswk@ifi.uio.no (University)
christopher.kolstad@gmail.com (Home)
chriswk@ovitas.no (Job)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message