lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Thu, 11 Sep 2008 05:12:08 GMT
You can't hold the ThreadLocal value in a WeakReference, because  
there is no hard reference between enumeration calls (so it would be  
cleared out from under you while enumerating).

All of this occurs because you have some objects (readers/segments  
etc.) that are shared across all threads, but these contain objects  
that are 'thread/search state' specific. These latter objects are  
essentially "cached" for performance (so you don't need to seek and  
read, sequential buffer access, etc.)

A sometimes better solution is to have the state returned to the  
caller, and require the caller to pass/use the state later - then you  
don't need thread locals.

You can accomplish a similar solution by returning a "SessionKey"  
object, and have the caller pass this later.  You can then have a  
WeakHashMap of SessionKey,SearchState that the code can use.  When  
the SessionKey is destroyed (no longer referenced), the state map can  
be cleaned up automatically.



On Sep 10, 2008, at 11:30 PM, Noble Paul നോബിള്‍  
नोब्ळ् wrote:

> When I look at the reference tree That is the feeling I get. if you
> held a WeakReference it would get released .
>  |- base of org.apache.lucene.index.CompoundFileReader$CSIndexInput
>               |- input of org.apache.lucene.index.SegmentTermEnum
>                   |- value of java.lang.ThreadLocal$ThreadLocalMap 
> $Entry
>
> On Wed, Sep 10, 2008 at 8:39 PM, Chris Lu <chris.lu@gmail.com> wrote:
>> Does this make any difference?
>> If I intentionally close the searcher and reader failed to release  
>> the
>> memory, I can not rely on some magic of JVM to release it.
>> --
>> Chris Lu
>> -------------------------
>> Instant Scalable Full-Text Search On Any Database/Application
>> site: http://www.dbsight.net
>> demo: http://search.dbsight.com
>> Lucene Database Search in 3 minutes:
>> http://wiki.dbsight.com/index.php? 
>> title=Create_Lucene_Database_Search_in_3_minutes
>> DBSight customer, a shopping comparison site, (anonymous per  
>> request) got
>> 2.6 Million Euro funding!
>>
>> On Wed, Sep 10, 2008 at 4:03 AM, Noble Paul  
>> നോബിള്‍ नोब्ळ्
>> <noble.paul@gmail.com> wrote:
>>>
>>> Why do you need to keep a strong reference?
>>> Why not a WeakReference ?
>>>
>>> --Noble
>>>
>>> On Wed, Sep 10, 2008 at 12:27 AM, Chris Lu <chris.lu@gmail.com>  
>>> wrote:
>>>> The problem should be similar to what's talked about on this  
>>>> discussion.
>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal
>>>>
>>>> There is a memory leak for Lucene search from Lucene-1195.(svn  
>>>> r659602,
>>>> May23,2008)
>>>>
>>>> This patch brings in a ThreadLocal cache to TermInfosReader.
>>>>
>>>> It's usually recommended to keep the reader open, and reuse it when
>>>> possible. In a common J2EE application, the http requests are  
>>>> usually
>>>> handled by different threads. But since the cache is  
>>>> ThreadLocal, the
>>>> cache
>>>> are not really usable by other threads. What's worse, the cache  
>>>> can not
>>>> be
>>>> cleared by another thread!
>>>>
>>>> This leak is not so obvious usually. But my case is using  
>>>> RAMDirectory,
>>>> having several hundred megabytes. So one un-released resource is  
>>>> obvious
>>>> to
>>>> me.
>>>>
>>>> Here is the reference tree:
>>>> org.apache.lucene.store.RAMDirectory
>>>>  |- directory of org.apache.lucene.store.RAMFile
>>>>      |- file of org.apache.lucene.store.RAMInputStream
>>>>          |- base of
>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput
>>>>              |- input of org.apache.lucene.index.SegmentTermEnum
>>>>                  |- value of java.lang.ThreadLocal$ThreadLocalMap 
>>>> $Entry
>>>>
>>>>
>>>> After I switched back to svn revision 659601, right before this  
>>>> patch is
>>>> checked in, the memory leak is gone.
>>>> Although my case is RAMDirectory, I believe this will affect  
>>>> disk based
>>>> index also.
>>>>
>>>> --
>>>> Chris Lu
>>>> -------------------------
>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>> site: http://www.dbsight.net
>>>> demo: http://search.dbsight.com
>>>> Lucene Database Search in 3 minutes:
>>>>
>>>> http://wiki.dbsight.com/index.php? 
>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>> DBSight customer, a shopping comparison site, (anonymous per  
>>>> request)
>>>> got
>>>> 2.6 Million Euro funding!
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>>
>
>
>
> -- 
> --Noble Paul


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message