lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Wed, 10 Sep 2008 00:03:40 GMT

Chris Lu wrote:

> The problem should be similar to what's talked about on this  
> discussion.
> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal

The "rough" conclusion of that thread is that, technically, this isn't  
a memory leak but rather a "delayed freeing" problem.  Ie, it may take  
longer, possibly much longer, than you want for the memory to be freed.

> There is a memory leak for Lucene search from Lucene-1195.(svn  
> r659602, May23,2008)
>
> This patch brings in a ThreadLocal cache to TermInfosReader.

One thing that confuses me: TermInfosReader was already using a  
ThreadLocal to cache the SegmentTermEnum instance.  What was added in  
this commit (for LUCENE-1195) was an LRU cache storing Term ->  
TermInfo instances.  But it seems like it's the SegmentTermEnum  
instance that you're tracing below.

> It's usually recommended to keep the reader open, and reuse it when
> possible. In a common J2EE application, the http requests are usually
> handled by different threads. But since the cache is ThreadLocal,  
> the cache
> are not really usable by other threads. What's worse, the cache can  
> not be
> cleared by another thread!
>
> This leak is not so obvious usually. But my case is using  
> RAMDirectory,
> having several hundred megabytes. So one un-released resource is  
> obvious to
> me.
>
> Here is the reference tree:
> org.apache.lucene.store.RAMDirectory
>  |- directory of org.apache.lucene.store.RAMFile
>      |- file of org.apache.lucene.store.RAMInputStream
>          |- base of org.apache.lucene.index.CompoundFileReader 
> $CSIndexInput
>              |- input of org.apache.lucene.index.SegmentTermEnum
>                  |- value of java.lang.ThreadLocal$ThreadLocalMap 
> $Entry

So you have a RAMDir that has several hundred MB stored in it, that  
you're done with yet through this path Lucene is keeping it alive?

Did you close the RAMDir?  (which will null its fileMap and should  
also free your memory).

Also, that reference tree doesn't show the ThreadResources class that  
was added in that commit -- are you sure this reference tree wasn't  
before the commit?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message