lucene-java-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-java Wiki] Update of "ImproveSearchingSpeed" by MikeMcCandless
Date Sun, 10 Jun 2007 09:02:28 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-java Wiki" for change notification.

The following page has been changed by MikeMcCandless:
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed

New page:
= How to make searching faster =

Here are some things to try to speed up the seaching speed of your Lucene application.  Please
see ImproveIndexingSpeed for how to speed up indexing.

 * '''Be sure you really need to speed things up.'''

 Many of the ideas here are simple to try, but others will necessarily add some complexity
to your application.  So be sure your searching speed is indeed too slow and the slowness
is indeed within Lucene.

 * '''Make sure you are using the latest version of Lucene.'''

 * '''Use a local filesystem.'''

 Remote filesystems are typically quite a bit slower for searching.  If the index must be
remote, try to mount the remote filesystem as a "readonly" mount.  In some cases this could
improve performance.

 * '''Get faster hardware, especially a faster IO system.'''

 * '''Add RAM to your hardware and/or increase the heap size for the JVM.'''

 For a large index, searching can use alot of RAM.  If you don't have enough RAM or your JVM
is not running with a large enough HEAP size then the JVM can hit swapping and thrashing at
which point everything will run slowly.

 * '''Use one instance of [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html
IndexSearcher].'''

 Share a single [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html
IndexSearcher] across queries and across threads in your application.

 * '''Re-open the [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html
IndexSearcher] only when necessary.'''

 You must re-open the [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html
IndexSearcher] in order to see make newly committed changes visible to searching.  However,
doing this is costly because many caches are lost when opening a new searcher. Consider using
a so called [http://wiki.apache.org/solr/SolrCaching warming] technique which allows the searcher
to warm up its caches before the first query hits.

 * '''Run [http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexWriter.html#optimize()
optimize] on your index before searching.'''

 An optimized index has only 1 segment to search which can be much faster than the many segments
that will normally be created, especially for a large index.  If your application does not
often update the index then it pays to build the index, optimize it, and use the optimized
one for searching.  If instead you are frequently updating the index and then refreshing searchers,
then optimizing will likely be too costly and you should decrease mergeFactor instead.

 * '''Decrease [http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexWriter.html#setMergeFactor(int)
mergeFactor].'''

 Smaller mergeFactors mean fewer segments and searching will be faster.  However, this will
slow down indexing speed, so you should test values to strike an appropriate balance for your
application.

 * '''Limit usage of stored fields and term vectors.'''

 Retrieving these from the index is quite costly.  Typically you should only retrieve these
for the current "page" the user will see, not for all documents in the full result set.  For
each document retrieved, Lucene must seek to a different location in various files.  Try sorting
the documents you need to retrieve by docID order first.

 * '''Use [http://lucene.apache.org/java/docs/api/org/apache/lucene/document/FieldSelector.html
FieldSelector] to carefully pick which fields are loaded, and how they are loaded, when you
retrieve a document.'''

Mime
View raw message