lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Volodymyr Bychkoviak <vbychkov...@i-hypergrid.com>
Subject Re: OutOfMemory error when searching
Date Fri, 19 Aug 2005 10:47:49 GMT
Hi,
IndexReader class has static method getCurrentVersion.
you can check index version and if was changed since last time then 
reopen IndexSearcher.

Peter Gelderbloem wrote:

>Hi,
>I foresee that I may run into similar problems as I am opening and
>closing IndexSearchers all over the place. 
>Unfortunately I do not see a way to keep the IndexSearcher open as
>indexing occurs at random times in my system and the searches need to be
>current.
>Any suggestions on how to reuse the IndexSearcher in this situation?
>
>Peter 
>-----Original Message-----
>From: Doug Cutting [mailto:cutting@apache.org] 
>Sent: 18 August 2005 19:09
>To: java-user@lucene.apache.org
>Subject: Re: OutOfMemory error when searching
>
>Fredrik wrote:
>  
>
>>Opening the index with Luke, I can see the following:
>>Number of fields: 17
>>Number of documents: 1165726
>>Number of terms: 6721726
>>
>>The size of the index is approx 5,3 GB.
>>Lucene version is 1.4.3.
>>
>>The index contains Norwegian terms, but lots of inline HTML, etc
>>is probably increasing the index term count (should 'wash' these
>>unwanted terms away when indexing documents). The analysis below
>>shows that TermInfosReader.java:132 -> get() is trying to allocate
>>a huge memory slab.
>>    
>>
>[ ... ]
>  
>
>>'need 532676624 bytes' means that something is allocating a 500Mb slab
>>of memory.
>>    
>>
>
>Lucene will try to allocate an array of 6721726/128 ~= 50k terms.  The 
>array alone will require a 200kB "slab" and the terms perhaps 1MB or 
>more.  But not 500MB.  So I think something else is the culprit here.
>
>Have you tried inserting print statements at the suspected allocations, 
>to see how big the arrays actually are?  Are you perhaps creating a new 
>IndexSearcher per search, rather than reusing a single IndexSearcher? 
>1MB per query could quickly exhaust RAM if the GC can't keep up.
>
>Doug
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>  
>

-- 
regards,
Volodymyr Bychkoviak


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message