lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lixin Meng" <li...@fulldegree.com>
Subject RE: parallizing index building
Date Mon, 30 Jun 2003 19:11:20 GMT
Where can I find any sample code or documentation about merging a set of
small indexes into one big index?

Lixin

-----Original Message-----
From: Doug Cutting [mailto:cutting@lucene.com]
Sent: Monday, June 30, 2003 10:24 AM
To: Lucene Users List
Subject: Re: parallizing index building


Marc Dumontier wrote:
> I'm indexing 500 XML files each ~150Mb on an 8 CPU machine.
>
> I'm wondering what the best strategy for making maximum use of resources
is. I have the tweaked the single process indexer to index 5000 records (not
files) in memory before writing out to disk.
>
> Should i create an IndexThread and share the IndexWriter object across 5
threads..then monitor when one ends to start another, etc. Or should i
create difference indexes then to a series of merges.

Creating multiple indexes in parallel and then merging them at the end
will probably be fastest.

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message