Mailing-List: contact user-help@lucenenet.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@lucenenet.apache.org
MIME-Version: 1.0
References: <CABYArMNNsodGozKX+gkOU66x6OkpivO8o5qi60Y_Wr+_fMuWRg@mail.gmail.com>
In-Reply-To: <CABYArMNNsodGozKX+gkOU66x6OkpivO8o5qi60Y_Wr+_fMuWRg@mail.gmail.com>
From: JA Purcell <9urc311@gmail.com>
Date: Wed, 27 Sep 2017 13:28:55 +0000
Message-ID: <CAPjcC6SKysbsm6e2Ce_1_HRt737J80pLyddauEr2ruxG_CTmtQ@mail.gmail.com>
Subject: Re: confused about segment merging and commits
To: user@lucenenet.apache.org
Content-Type: multipart/alternative; boundary="001a114dd2fe9e3f51055a2bc6a4"
archived-at: Wed, 27 Sep 2017 13:29:14 -0000

--001a114dd2fe9e3f51055a2bc6a4
Content-Type: text/plain; charset="UTF-8"

Hey Jonathan,

I can only speak from my experience with Lucene.Net, but the XML doc
comment on the Commit method states that it does not wait for any
background merges to finish.  There is a WaitForMerges method on the
IndexWriter if you wanted to wait for those to finish.  From the way I
understand it, segments are still part of the searchable index; so the
reader would still read them in a search.  You merge them together to
reduce the number of files that the reader must parse reducing search
time.

If you have access to the Lucene In Action book, section 2.13 goes into
depth with how most of this works.  The XML docs are pretty good too.

I've never used it, but you can also have the IndexWriter write it's
verbose logging to a file via the SetInfoStream method so that you can see
exactly what it's doing.

Hope that helps some,
Adam

On Wed, Sep 27, 2017 at 5:59 AM Jonathan Resnick <jresnick@gmail.com> wrote:

> Hi,
>
> I am trying to understand how segment merging interacts with commits.
>
> Consider the following timeline of events:
>
> 1. IndexWriter is opened on an index.
> 2. IndexWriter is used to add/update/delete docs, but not yet commit the
> changes.
> 3. Activity in step 2 triggers segment merging on a background thread.
> 4. Commit() is called on IndexWriter while merging in step 3 has not yet
> finished.
>
> Does the Commit() in step 4 block while the segment merge in step 3
> finishes?
> If not, then when is the segment merge in 3 "committed" to the index? (i.e.
> at what point would a new IndexReader see the merged segment file?)
> Or does segment merging happen entirely independently of commits?
>
> [More context: we are trying to build a backup system that copies the index
> files to a backup server after every commit. Initially I thought it would
> be sufficient to just keep track of file add/update/deletes since the
> previous commit, but if segment merging is happening concurrently then
> perhaps it's not so simple?]
>
> More generally, is there any in-depth documentation available describing
> how segment merging interacts with commits (even if it's for the Java
> version of Lucene)?  My web searches have not turned up much...
>
> Many thanks,
> Jonathan
>

--001a114dd2fe9e3f51055a2bc6a4--