lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lixin Meng" <>
Subject RE: parallizing index building
Date Mon, 30 Jun 2003 19:11:20 GMT
Where can I find any sample code or documentation about merging a set of
small indexes into one big index?


-----Original Message-----
From: Doug Cutting []
Sent: Monday, June 30, 2003 10:24 AM
To: Lucene Users List
Subject: Re: parallizing index building

Marc Dumontier wrote:
> I'm indexing 500 XML files each ~150Mb on an 8 CPU machine.
> I'm wondering what the best strategy for making maximum use of resources
is. I have the tweaked the single process indexer to index 5000 records (not
files) in memory before writing out to disk.
> Should i create an IndexThread and share the IndexWriter object across 5
threads..then monitor when one ends to start another, etc. Or should i
create difference indexes then to a series of merges.

Creating multiple indexes in parallel and then merging them at the end
will probably be fastest.


To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message