lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <>
Subject Re: Concurrent merge
Date Sat, 03 Mar 2007 16:30:51 GMT
I think it'd be interesting to see how much better this is and in what kinds of scenarios it
makes things faster, so if this is going to be separate from IndexWriter, why not try it.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy --  -  Tag  -  Search  -  Share

----- Original Message ----
From: Ning Li <>
Sent: Friday, March 2, 2007 5:44:55 PM
Subject: Re: Concurrent merge

Many good points! Thanks, guys!

When background merge is employed, document additions can
out-pace merging, no matter how many background merge threads
are used. Blocking has to happen at some point.

So, if we do anything, we make it simple. I agree with what
Robert and Yonik have proposed: documents can be buffered
while segments are merged, but no more than maxBufferedDocs
can be buffered at any time.

Nadav asks a good question: if document additions can still
block, what is the point?

I see the following benefits:
1 Performance improvement to certain extent
2 Shorter ingestion hiccups: Especially when the ingestion
   rate is low, the hiccups can be very short. In the current
   single-threaded IndexWriter, the hiccups can be very long
   when a document ingestion triggers a cascade of merges.

Doron has a valid concern: Lucene's being single threaded is
a simplifying factor.

Agree. That's why I propose this as a subclass of IndexWriter,
not to replace the current single-threaded IndexWriter.

So, given this simple background merge design, are people
interested in the benefits it provides?

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message