lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Waite <p...@waite.net.nz>
Subject Re: index architectures
Date Thu, 19 Oct 2006 01:22:30 GMT
Some excellent feedback guys - thanks heaps.


On my OOM issue, I think Hoss has nailed it here:

> That said: if you are seeing OOM errors when you sort by a field (but
> not when you use the docId ordering, or sort by score) then it sounds
> like  you are keeping refrences to IndexReaders arround after you've
> stoped using them -- the FieldCache is kept in a WeakHashMap keyed off of
> hte IndexReader, so it should get garbage collected as soon sa you let go
> of it.  Another possibility is that you are sorting on too many fields
> for it to be able to build the FieldCache for all of them in the RAM you
> have available. 


I'm using a piece of code written by Peter Halacsy which implements a
SearcherCache class. When we do a search we request a searcher, and this
class looks after giving us one.

It checks whether the index has been updated since the most recent Searcher
was created. If so it creates a new one.

At the same time it 'retires' outdated Searchers, once they have no queries
busy with them.

Looking at that code, if the system gets busy indexing new stuff, and doing
complex searches this is all rather open-ended as to the potential number
of fresh Searchers being created, each with the overhead of building its
FieldCache for the first time. No wonder I'm having problems as the archive
has grown! Looking at it in this light, my OOM's all seem to come just
after a bout of articles have been indexed, and querying is being done
simultaneously, so it does fit.

I guess a solution is probably to cap this process with a maximum number
of active Searchers, meaning potentially some queries might be fobbed off
with slightly out of date versions of the index, in extremis, but it would
right itself once everything settles down again.

Obviously the index partitioning would probably make this a non-issue, but
it seems better to sort the basic problem out anyway, and make it 100%
stable.

Thanks Hoss!

Cheers,
Paul.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message