lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Merging indexes & multicore/multithreading
Date Tue, 02 Dec 2008 19:05:34 GMT

With ConcurrentMergeScheduler, adding all indexes at once to a single  
IndexWriter will use multiple threads to do the merging, assuming you  
have enough total segments that need merging (> 2 X mergeFactor will  
use 2 threads; > 3 X mergeFactor will use 3, etc.; CMS defaults to max  
3 merge threads at once).

I think merging is mainly IO bound, so you may only see a win using  
multiple threads if your IO system has concurrency to exploit.

Try both and post back the results :)

Mike

Glen Newton wrote:

> Let's say I have 8 indexes on a 4 core system and I want to merge them
> (inside a single vm instance).
> Is it better to do a single merge of all 8, or to in parallel threads
> merge in pairs, until there is only a single index left? I guess the
> question involves how multi-threaded merging is and if it will take
> advantage of all cores.
>
> thanks,
>
> -glen
>
> -- 
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message