lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: Concurrent merge
Date Wed, 21 Feb 2007 17:55:09 GMT
The point I was trying to make is that even if you use a background  
thread for merging, that at some point it may block, if the document  
additions out-pace the merging.

On Feb 21, 2007, at 11:47 AM, Ning Li wrote:

> I agree that the current blocking model works for some applications,
> especially if the indexes are batch built.
>
> But other applications, e.g. with online indexes, would greatly
> benefit from a non-blocking model. Most systems that merge data
> support background merges. As long as we keep it simple (how about the
> original proposal?), applications will benefit from this.
>
> Ning
>
> On 2/20/07, Mike Klaas <mike.klaas@gmail.com> wrote:
>> On 2/20/07, robert engels <rengels@ix.netcom.com> wrote:
>> > What about a queue of segments to merge. The add document will add
>> > segments to the queue, if the queue contains too many segments it
>> > blocks.
>> >
>> > Another thread reads the segments from the queue and merges them.
>> >
>> > This would effectively block adding of documents some times, but  
>> that
>> > is not different than what happens now.
>>
>> In our custom database (that has a lucene-like logarithmic set of
>> segmented databases), we have a separate merge thread controlled with
>> a bounded queue as you suggest.  As you note, the unbounded wait time
>> on doc addition must still be present if you have limits on how far
>> ahead you can let the doc addition run ahead of the merging.
>>
>> You could imagine more complexity, allowing smaller sub-merges to
>> occur of fresh segments while large background merges were  
>> proceeding.
>>
>> For us, ironing out the resulting complexity wasn't a beneficial use
>> of time, and in retrospect I prefer lucene's model.
>>
>> -Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message