lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Li" <ning.li...@gmail.com>
Subject Re: Concurrent merge
Date Fri, 30 Mar 2007 02:48:44 GMT
FYI: Patch submitted in http://issues.apache.org/jira/browse/LUCENE-847.

Cheers,
Ning

"Here is a patch for concurrent merge as discussed in:
http://www.gossamer-threads.com/lists/lucene/java-dev/45651?search_string=concurrent%20merge;#45651

"I put it under this issue because it helps design and verify a
factored merge policy which would provide good support for concurrent
merge.

"As described before, a merge thread is started when a writer is
created and stopped when the writer is closed. The merge process
consists of three steps: first, create a merge task/spec; then, carry
out the actual merge; finally, "commit" the merged segment (replace
segments it merged in segmentInfos), but only after appropriate
deletes are applied. The first and last steps are fast and
synchronous. The second step is where concurrency is achieved. Does it
make sense to capture them as separate steps in the factored merge
policy?

"As discussed in
http://www.gossamer-threads.com/lists/lucene/java-dev/45651?search_string=concurrent%20merge;#45651:
documents can be buffered while segments are merged, but no more than
maxBufferedDocs can be buffered at any time. So this version provides
limited concurrency. The main goal is to achieve short ingestion
hiccups, especially when the ingestion rate is low. After the factored
merge policy, we could provide different versions of concurrent merge
policies which provide different levels of concurrency. :-)"


On 3/3/07, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
> I think it'd be interesting to see how much better this is and in what kinds of scenarios
it makes things faster, so if this is going to be separate from IndexWriter, why not try it.
>
> Otis
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message