Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <1FA15019-4090-4736-8CAF-89275F5735E5@mikemccandless.com>
From: Michael McCandless <lucene@mikemccandless.com>
To: java-user@lucene.apache.org
In-Reply-To: <5e76f3840812021017m4c33ad7fj919a3edf466b90ba@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: Merging indexes & multicore/multithreading
Date: Tue, 2 Dec 2008 14:05:34 -0500
References: <5e76f3840812021017m4c33ad7fj919a3edf466b90ba@mail.gmail.com>


With ConcurrentMergeScheduler, adding all indexes at once to a single  
IndexWriter will use multiple threads to do the merging, assuming you  
have enough total segments that need merging (> 2 X mergeFactor will  
use 2 threads; > 3 X mergeFactor will use 3, etc.; CMS defaults to max  
3 merge threads at once).

I think merging is mainly IO bound, so you may only see a win using  
multiple threads if your IO system has concurrency to exploit.

Try both and post back the results :)

Mike

Glen Newton wrote:

> Let's say I have 8 indexes on a 4 core system and I want to merge them
> (inside a single vm instance).
> Is it better to do a single merge of all 8, or to in parallel threads
> merge in pairs, until there is only a single index left? I guess the
> question involves how multi-threaded merging is and if it will take
> advantage of all cores.
>
> thanks,
>
> -glen
>
> -- 
>
> -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org