lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Stoppelman" <stop...@gmail.com>
Subject Re: replication question
Date Tue, 16 Dec 2008 20:32:49 GMT
Hi Yonik,

Thanks for the response.

reply inline.

On Tue, Dec 16, 2008 at 6:44 AM, Yonik Seeley <yseeley@gmail.com> wrote:

> On Tue, Dec 16, 2008 at 1:04 AM, Michael Stoppelman <stopman@gmail.com>
> wrote:
> > I've got a question from Doug's original email about replication (
> > http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg12709.html
> ):
> >
> > "1. On the index master, periodically checkpoint the index. Every minute
> or
> > so the IndexWriter is closed and a 'cp -lr index index.DATE' command is
> > executed from Java, where DATE is the current date and time. This
> > efficiently makes a copy of the index when its in a consistent state by
> > constructing a tree of hard links. If Lucene re-writes any files (e.g.,
> the
> > segments file) a new inode is created and the copy is unchanged."
> >
> > Is closing the IndexWriter really a requirement on taking a snapshot? Or
> can
> > one take a snapshot on an index being written, I've done this in my
> > development environment and it seems to work fine w/o closing the
> > IndexWriter.
>
> There are subtle race conditions if you try to do this with a changing
> index.
> At any instance in time, the index should be consistent, *but* you
> can't actually make a snapshot instantaneously.


Is the race condition in writing out the segments.gen or segments_N files?
>From my understanding index segments once closed by the IndexWriter they
aren't modified again (they might be deleted though if they're merged away).


>
> So this is doable, but it would require some complex retry logic like
> IndexReader has when opening an index.
>
> > Also the solr replication shell scripts don't seem to worry
> > about this either.
>
> Solr takes snapshots when it knows it's not updating the index (new
> index changes are internally blocked when calling snapshooter).
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message