I think it'd be interesting to see how much better this is and in what kinds of scenarios it
makes things faster, so if this is going to be separate from IndexWriter, why not try it.
Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/ - Tag - Search - Share
----- Original Message ----
From: Ning Li <ning.li.li@gmail.com>
To: java-dev@lucene.apache.org
Sent: Friday, March 2, 2007 5:44:55 PM
Subject: Re: Concurrent merge
Many good points! Thanks, guys!
When background merge is employed, document additions can
out-pace merging, no matter how many background merge threads
are used. Blocking has to happen at some point.
So, if we do anything, we make it simple. I agree with what
Robert and Yonik have proposed: documents can be buffered
while segments are merged, but no more than maxBufferedDocs
can be buffered at any time.
Nadav asks a good question: if document additions can still
block, what is the point?
I see the following benefits:
1 Performance improvement to certain extent
2 Shorter ingestion hiccups: Especially when the ingestion
rate is low, the hiccups can be very short. In the current
single-threaded IndexWriter, the hiccups can be very long
when a document ingestion triggers a cascade of merges.
Doron has a valid concern: Lucene's being single threaded is
a simplifying factor.
Agree. That's why I propose this as a subclass of IndexWriter,
not to replace the current single-threaded IndexWriter.
So, given this simple background merge design, are people
interested in the benefits it provides?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
|