lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <>
Subject Re: Concurrent merge
Date Wed, 21 Feb 2007 23:23:26 GMT
On 2/21/07, Doron Cohen <> wrote:
> Ning Li wrote:
> > There are three main challenges in enabling concurrent merge:
> >   1 a robust merge policy
> >   2 detect when merge lags document additions/deletions
> >   3 how to slow down document additions/deletions (and amortize
> >     the cost) when merge falls behind
> I wonder what it means for current API semantics -
> - An application today can set max-bufferred-docs to N, and after
> the Nth (or N+1th?) call to addDoc returns, a newly opened searcher
> would see these docs. With merges in a background thread this
> might not hold.
> - Today, after add(), an application can call flush() or close(),
> but with a background merge thread these calls would be blocked.
> Mmm... this is probably not a behavior change, because today
> these operations can trigger a merge that would take a long(er) time.

We shouldn't advertise or guarantee that behavior.  This wasn't even
true before the new merge policy was implemented.

> - numRamDocs() and ramSizeInBytes() - not sure what they mean
> once a background merge thread had started.

IMO, for the current "batch" of documents being buffered.
The "old" buffered documents should be flushed to disk ASAP.

> Still, having non blocking adds is compelling.

Somewhat... It would result in some performance increase...
overlapping analysis of new documents with merging of other segments,
resulting in a higher CPU utilization (esp on multi-processor
systems).  The larger the maxBufferedDocs, the better.

The downside is another complexity increase though.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message