lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lutz Fechner <>
Subject RE: Lucene Searcher Caching and Performance
Date Wed, 05 Aug 2015 10:18:16 GMT

not sure what Lucene Version you are using and how you cache you Readers.

But you can adjust memory consumption per reader afaik by setings "TermInfos index divisor"
to something larger 1 when opening the Searcher.
At least that works for 2.9.4.

" it takes a little while for the server to return to normal operating limits thereafter "
- Did you check your VM behavior? 
Just adding more mem to it might not always work (or even make the problem worse) - depending
on VM (vendor and version), and GC in place. 



-----Original Message-----
From: kiwi clive [] 
Sent: Dienstag, 4. August 2015 17:41
To: Java-user
Subject: Lucene Searcher Caching and Performance

Hi Guys,
We have an index/query server that contains several thousand fairly hefty indexes. Each searcher
is shared between many 'user-threads' and once opened we keep the searcher in a cache which
is refreshed depending on how often it is used. Due to memory limitations on the server, we
need some kind of LRU mechanism to drop unused searchers to make way for newer ones.
We are seeing load spikes when we get hit by queries that try to open several non-cached searches
at the same (or at least a small delta) time. This looks to be the disks struggling to open
all the appropriate files for that period, and it takes a little while for the server to return
to normal operating limits thereafter.
Given that upgrading hardware/memory is not currently an option, we need a way to smooth over
these spikes, even if it is at the cost of slowing query performance overall. 

It strikes me that if we could cache all of our searchers on the machine (ie have all of our
indexes 'open for business'), possibly having to alter kernel parameters to cater for the
large number of file handles, without caching many query results, this might solve the problem,
without pushing memory usage too high. Also, the higher number of searchers stored in the
heap is going to steal space from the lucene filecache so is there a recommended mechanism
for doing this?
So is there a way to mimimize the searcher cache memory footprint to possibly keep more of
them in memory, at the cost of storing less data?
Any insight would be most appreciated.

View raw message