lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Indexing speed on NTFS
Date Tue, 31 May 2011 07:30:25 GMT
On Tue, 2011-05-31 at 08:52 +0200, Maciej Klimczuk wrote:
> I did some testing with 3.1.0 demo on Windows and encountered some strange  
> bahaviour. I tried to index ~60000 small text documents using the demo.
> - First trial took about 18 minutes.
> - Second and third trial took about 2 minutes.

First trial sounds strange, even if the documents are single files and
on a traditional harddisk. The 500 documents/second in the subsequent
trials sounds okay for small documents.

> [...] I repeated this with 30B, 60MB and 100MB, but all the time I
> aborted the process and removed index, it was recreated to the
> previous size in a matter of tens of seconds (less than minute),
> and after that it was growing slowly.

It seems like you have extremely slow read access from your storage and
a small enough data set so that the generated index is still in the
write buffer.

Are you perhaps using Windows XP? It drops back to PIO-mode under some
circumstances and it really hurts performance. You can read about it at
http://winhlp.com/node/10

> If there is a document or site explaining this or it was asked before,  
> please forgive me; just searching about Lucene indexing performance on  
> NTFS doesn't help me much...

There should not be any problems like the one you describe with NTFS.
I have used Windows XP with NTFS myself for a year or two and did not
encounter anything like it.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message