lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Unable to improve performance
Date Wed, 01 Apr 2009 10:04:15 GMT
On Fri, 2009-03-27 at 12:07 +0100, Paul Taylor wrote:

[2Gb index, 7 million documents(?)]

> I ran the test a number of times with 30 threads, and max memory of 
> 3500mb I was processing 10,000 records in about 43 seconds ( 233 
> queries/second) , the index was stored on a solid state drive running on 
> a MacBook Pro (2.66 Ghz Intel Core 2 Duo, 4GB DDR). I dont really have a 
> view on whether this is a good result or not but I was keen to try a few 
> other things to see if I could improve performance further, but all my 
> efforts have had minimal effect.

You might want to try reducing the number of threads all the way down to
3 or 4 and queue pending searches instead, but I doubt it will change
much - as far as I know, the SSDs in MacBooks are quite okay with regard
to read-latency and with such a small index the system will probably
cache most of it anyway.

I can see elsewhere that you have upped the speed to 466 q/sec by
switching to NIOFSDirectory, so my guess is that you're now CPU and
memory speed bound.

You could try the freeware tool visualVM that profiles running Java
applications. It is extremely easy to use (just run it and select your
application from a list) and it will show you where the CPU-time is
used. Of course, if you're just using simple query analysis with Lucene
supplied Analyzers, there's probably not much you can do about it. On
the other hand, it might show you that you're spending a lot of time
generating queries or similar outside-Lucene-work.

> The main thing Im suprised about is I was expecting a massive difference 
> in holding the index in memory instead on disk

Solid state Drives (and disk cache) rules. Our experiments shows very
little performance increase going from SSD to RAMDirectory.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message