lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Gray <Stephen.G...@anu.edu.au>
Subject Re: Memory leak (JVM 1.6 only)
Date Fri, 18 May 2007 05:40:01 GMT
Hi Otis,

Thanks very much for your reply.

I've removed the LuceneIndexAccessor code, and still have the same 
problem, so that at least rules out LuceneIndexAccessor as the source. 
maxBufferedDocs is just set to the default, which I believe is 10.

I've tried jconsole, + jmap/jhat for looking at the contents of the 
heap. One interesting thing is that although the memory allocated as 
reported by the processes tab of Windows Task Manager goes up and up, 
and the JVM eventually crashes with an OutOfMemory error, the total size 
of heap + non-heap as reported by jconsole is constant and much lower 
than the Windows-reported allocated memory. I've also tried Netbeans 
profiler, which suggests that the variables in the heap that are 
continually surviving garbage collection do not all originate from one 
class.

I can't definitely rule out Tomcat. Clearly something is interacting 
with a change in JVM 1.6 and causing the problem. The fact that it only 
occurs during indexing not searching suggested that it might be related 
to the indexing code rather than Tomcat. It's much more likely that it's 
my code than Lucene, but I can't see anything in my code though I'm 
definitely no expert on memory leaks. All the variables created during 
indexing except IndexReader and Searcher instances are local to my 
addDocument function so should be garbage collected after each document 
is added. I did wonder if it might be related to SnowballAnalyzer as 
quite a few long lived variables in the heap were created by this - but 
then the heap is not increasing.

Regards,
Steve

Otis Gospodnetic wrote:
> Hi Steve,
>
> You said the OOM happens only when you are indexing.  You don't need LuceneIndexAccess
for that, so get rid of that to avoid one suspect that is not part of Lucene core.  What is
your maxBufferedDocs set to?  And since you are using JVM 1.6, check out jmap, jconsole &
friends, they'll provide insight into where your OOM is coming from.  I see your app is a
webapp.  How do you know it's Lucene and its indexing that are the source of OOM and not something
else, such as a bug in Tomcat?
>
> Otis
>  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>
> ----- Original Message ----
> From: Stephen Gray <Stephen.Gray@anu.edu.au>
> To: java-user@lucene.apache.org
> Sent: Tuesday, May 15, 2007 2:31:05 AM
> Subject: Memory leak (JVM 1.6 only)
>
> Hi everyone,
>
> I have an application that indexes/searches xml documents using Lucene. 
> I'm having a problem with what looks like a memory leak, which occurs 
> when indexing a large number of documents, but only when the application 
> is running under JVM 1.6. Under JVM 1.5 there is no problem. What 
> happens is that the memory allocated consistently rises during indexing 
> until the JVM crashes with an OutOfMemory exception.
>
> I'm using Lucene 2.1, and am using Maik Schreiber's LuceneIndexAccess 
> API, which hands out references to cached IndexWriter/Reader/Searchers 
> to objects that need to use them, and handles closing and re-opening 
> IndexSearchers after documents are added to the index. The application 
> is running under Tomcat 6.
>
> I'm a bit out of my depth determining the source of the leak - I've 
> tried using Netbeans profiler, which shows a large number of HashMap 
> instances that survive a long time, but these are created by many 
> different classes so it's difficult to pinpoint one source.
>
> Has anyone found similar problems with Lucene indexing operations 
> running under JVM 1.6? Does anyone have any suggestions re how to deal 
> with this?
>
> Any help much appreciated.
>
> Thanks,
> Steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


-- 
Stephen Gray
Archive IT Officer
Australian Social Science Data Archive
18 Balmain Crescent (Building #66)
The Australian National University
Canberra ACT 0200

Phone +61 2 6125 2185
Fax +61 2 6125 0627
Web http://assda.anu.edu.au/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message