lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@lucene.com>
Subject Re: parallizing index building
Date Mon, 30 Jun 2003 17:23:44 GMT
Marc Dumontier wrote:
> I'm indexing 500 XML files each ~150Mb on an 8 CPU machine.
> 
> I'm wondering what the best strategy for making maximum use of resources is. I have the
tweaked the single process indexer to index 5000 records (not files) in memory before writing
out to disk.
> 
> Should i create an IndexThread and share the IndexWriter object across 5 threads..then
monitor when one ends to start another, etc. Or should i create difference indexes then to
a series of merges.

Creating multiple indexes in parallel and then merging them at the end 
will probably be fastest.

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message