lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marc Dumontier" <mdumontier1...@rogers.com>
Subject parallizing index building
Date Fri, 27 Jun 2003 03:57:07 GMT
Hi,

I'm indexing 500 XML files each ~150Mb on an 8 CPU machine.

I'm wondering what the best strategy for making maximum use of resources is. I have the tweaked
the single process indexer to index 5000 records (not files) in memory before writing out
to disk.

Should i create an IndexThread and share the IndexWriter object across 5 threads..then monitor
when one ends to start another, etc. Or should i create difference indexes then to a series
of merges.

any help would be appreciated,

thanks,
Marc Dumontier
Bioinformatics Application Developer
Blueprint Initiative
Mount Sinai Hospital
Toronto
http://www.bind.ca

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message