lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Li" <ning.li...@gmail.com>
Subject Re: Concurrent merge
Date Wed, 21 Feb 2007 17:47:53 GMT
I agree that the current blocking model works for some applications,
especially if the indexes are batch built.

But other applications, e.g. with online indexes, would greatly
benefit from a non-blocking model. Most systems that merge data
support background merges. As long as we keep it simple (how about the
original proposal?), applications will benefit from this.

Ning

On 2/20/07, Mike Klaas <mike.klaas@gmail.com> wrote:
> On 2/20/07, robert engels <rengels@ix.netcom.com> wrote:
> > What about a queue of segments to merge. The add document will add
> > segments to the queue, if the queue contains too many segments it
> > blocks.
> >
> > Another thread reads the segments from the queue and merges them.
> >
> > This would effectively block adding of documents some times, but that
> > is not different than what happens now.
>
> In our custom database (that has a lucene-like logarithmic set of
> segmented databases), we have a separate merge thread controlled with
> a bounded queue as you suggest.  As you note, the unbounded wait time
> on doc addition must still be present if you have limits on how far
> ahead you can let the doc addition run ahead of the merging.
>
> You could imagine more complexity, allowing smaller sub-merges to
> occur of fresh segments while large background merges were proceeding.
>
> For us, ironing out the resulting complexity wasn't a beneficial use
> of time, and in retrospect I prefer lucene's model.
>
> -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message