lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dejan Nenov" <dejanne...@jollyobject.com>
Subject RE: Indexing large sets of documents?
Date Thu, 27 Jul 2006 19:11:12 GMT
Yes - parallelizing works great - we built a share-nothing java-spaces based
system at X1 and on a 11-way cluster were able to index 350 office documents
per second - this included the binary-2-text conversion, using Stellent INSO
libraries. The trick is to create separate indexes and, if you do not have a
federated search setup  - merge the indexes into one big index after they
are completed.

Dejan

-----Original Message-----
From: Michael J. Prichard [mailto:michael_prichard@mac.com] 
Sent: Thursday, July 27, 2006 9:30 AM
To: java-user@lucene.apache.org
Subject: Indexing large sets of documents?

I built an indexer that runs through email and its attachments, rips out 
content and what not and then creates a Document and adds it to an 
index.  It works w/ no problem.  The issue is that it takes around 3-5 
seconds per email and I have seen up to 10-15 seconds for email w/ 
attachments.  I need to index 750k emails and at those times it will 
take FOREVER!  I am trying to find places to cut a second or two here or 
there but are there any suggestions as to what I can do?  Should I look 
into parallelizing indexing?  Help?!

Thanks,
Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message