lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Li" <>
Subject Re: Concurrent merge
Date Fri, 02 Mar 2007 22:44:55 GMT
Many good points! Thanks, guys!

When background merge is employed, document additions can
out-pace merging, no matter how many background merge threads
are used. Blocking has to happen at some point.

So, if we do anything, we make it simple. I agree with what
Robert and Yonik have proposed: documents can be buffered
while segments are merged, but no more than maxBufferedDocs
can be buffered at any time.

Nadav asks a good question: if document additions can still
block, what is the point?

I see the following benefits:
1 Performance improvement to certain extent
2 Shorter ingestion hiccups: Especially when the ingestion
   rate is low, the hiccups can be very short. In the current
   single-threaded IndexWriter, the hiccups can be very long
   when a document ingestion triggers a cascade of merges.

Doron has a valid concern: Lucene's being single threaded is
a simplifying factor.

Agree. That's why I propose this as a subclass of IndexWriter,
not to replace the current single-threaded IndexWriter.

So, given this simple background merge design, are people
interested in the benefits it provides?

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message