lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Klaas" <>
Subject Re: Concurrent merge
Date Tue, 20 Feb 2007 20:52:27 GMT
On 2/20/07, robert engels <> wrote:
> What about a queue of segments to merge. The add document will add
> segments to the queue, if the queue contains too many segments it
> blocks.
> Another thread reads the segments from the queue and merges them.
> This would effectively block adding of documents some times, but that
> is not different than what happens now.

In our custom database (that has a lucene-like logarithmic set of
segmented databases), we have a separate merge thread controlled with
a bounded queue as you suggest.  As you note, the unbounded wait time
on doc addition must still be present if you have limits on how far
ahead you can let the doc addition run ahead of the merging.

You could imagine more complexity, allowing smaller sub-merges to
occur of fresh segments while large background merges were proceeding.

For us, ironing out the resulting complexity wasn't a beneficial use
of time, and in retrospect I prefer lucene's model.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message