lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <spr...@gmx.eu>
Subject RE: Swapping between indexes
Date Thu, 06 Mar 2008 17:22:06 GMT
> Since Lucene buffers in memory, you will always have the risk of
> losing recently added documents that haven't been flushed yet.
> Committing on every document would be too slow to be practical.

Well it is not sooo sloooow...

I have indexed 10.000 docs, resulting in 14 MB index. The index has 2 stored
fields and the tokenized content field.

With a commit after every add: 30 min.
With a commit after 100 add: 23 min.
Only one commit: 20 min.

(including time to get the document from the archive)

I use lucene 2.3 so a commit is a combination of closing and creating the
writer.
2.4/3.0 has a commit method which may be faster.

Before this test I thought it would be much slower than 30 min...

So one has to decide if correctness is more important than performance.

I use a batch size of 100, first committing lucene, then committing the
database which holds the status of the document if it is already indexed or
not.
If the db commit fails it is no problem, because my app does not care about
multiple indexed documents. But until now neither the lucene nor the db
commit ever failed...



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message