lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Gelderbloem" <Peter.Gelderbl...@mediasurface.com>
Subject RE: OutOfMemory error when searching
Date Fri, 19 Aug 2005 10:41:38 GMT

Hi,
I foresee that I may run into similar problems as I am opening and
closing IndexSearchers all over the place. 
Unfortunately I do not see a way to keep the IndexSearcher open as
indexing occurs at random times in my system and the searches need to be
current.
Any suggestions on how to reuse the IndexSearcher in this situation?

Peter 
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org] 
Sent: 18 August 2005 19:09
To: java-user@lucene.apache.org
Subject: Re: OutOfMemory error when searching

Fredrik wrote:
> Opening the index with Luke, I can see the following:
> Number of fields: 17
> Number of documents: 1165726
> Number of terms: 6721726
> 
> The size of the index is approx 5,3 GB.
> Lucene version is 1.4.3.
> 
> The index contains Norwegian terms, but lots of inline HTML, etc
> is probably increasing the index term count (should 'wash' these
> unwanted terms away when indexing documents). The analysis below
> shows that TermInfosReader.java:132 -> get() is trying to allocate
> a huge memory slab.
[ ... ]
> 'need 532676624 bytes' means that something is allocating a 500Mb slab
> of memory.

Lucene will try to allocate an array of 6721726/128 ~= 50k terms.  The 
array alone will require a 200kB "slab" and the terms perhaps 1MB or 
more.  But not 500MB.  So I think something else is the culprit here.

Have you tried inserting print statements at the suspected allocations, 
to see how big the arrays actually are?  Are you perhaps creating a new 
IndexSearcher per search, rather than reusing a single IndexSearcher? 
1MB per query could quickly exhaust RAM if the GC can't keep up.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message