lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Li" <>
Subject Re: Concurrent merge
Date Wed, 21 Feb 2007 17:47:53 GMT
I agree that the current blocking model works for some applications,
especially if the indexes are batch built.

But other applications, e.g. with online indexes, would greatly
benefit from a non-blocking model. Most systems that merge data
support background merges. As long as we keep it simple (how about the
original proposal?), applications will benefit from this.


On 2/20/07, Mike Klaas <> wrote:
> On 2/20/07, robert engels <> wrote:
> > What about a queue of segments to merge. The add document will add
> > segments to the queue, if the queue contains too many segments it
> > blocks.
> >
> > Another thread reads the segments from the queue and merges them.
> >
> > This would effectively block adding of documents some times, but that
> > is not different than what happens now.
> In our custom database (that has a lucene-like logarithmic set of
> segmented databases), we have a separate merge thread controlled with
> a bounded queue as you suggest.  As you note, the unbounded wait time
> on doc addition must still be present if you have limits on how far
> ahead you can let the doc addition run ahead of the merging.
> You could imagine more complexity, allowing smaller sub-merges to
> occur of fresh segments while large background merges were proceeding.
> For us, ironing out the resulting complexity wasn't a beneficial use
> of time, and in retrospect I prefer lucene's model.
> -Mike
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message