lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HODAC, Olivier" <olivier.ho...@airbus.com>
Subject Indexer large file and hi performance indexing
Date Wed, 06 Sep 2006 10:04:15 GMT

hello,

I design an application which bottleneck concerns the indexing process. Objects indexation
blocks the user's action. Furthermore, I need to index a large maount of documents (30000
per day) and save them on the file system.

The first developments have been initiate with lucene 1.4.3 and I had to create a cachedIndexer
which uses a RAMDirectory and a FS directory. For the searches, the cached indexer searches
in RAM and FS indexes. For deletion, the delete is done in the RAM and FS.

Each N documents to index in the cache (RAMDirectory), a thread dumps them to the FSDirectory,
with a synchronize.

In 1 month, I reached an index cfs file of 2.5Go, and the merge process (addIndexes) takes
5 minutes approx (solaris 5.9).!!!! => synchronize...

Questions:
* Is it a good way of doing things? 
* Does maxMergeDocs is a solution for the merge process to be lower?
* If I call addIndexes on the 2.5Go index file with a maxMergeDocs, it calls optimize. the
javadoc says that it calls optimize, which gathers to 1 file the indexdir. is it true whatever
the maxMergeDocs?

Thanks

 
This e-mail is intended only for the above addressee. It may contain
privileged information. If you are not the addressee you must not copy,
distribute, disclose or use any of the information in it. If you have
received it in error please delete it and immediately notify the sender.
Security Notice: all e-mail, sent to or from this address, may be
accessed by someone other than the recipient, for system management and
security reasons. This access is controlled under Regulation of
Investigatory Powers Act 2000, Lawful Business Practises.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message