lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Thu, 11 Sep 2008 09:56:40 GMT

What if we wrap the value in a WeakReference, but secondarily hold a  
hard reference to it in a "normal" list?

Then, when TermInfosReader is closed we clear that list of all its  
hard references, at which point GC will be free to reclaim the object  
out from under the ThreadLocal even before the ThreadLocal purges its  
stale entries.

Mike

robert engels wrote:

> You can't hold the ThreadLocal value in a WeakReference, because  
> there is no hard reference between enumeration calls (so it would be  
> cleared out from under you while enumerating).
>
> All of this occurs because you have some objects (readers/segments  
> etc.) that are shared across all threads, but these contain objects  
> that are 'thread/search state' specific. These latter objects are  
> essentially "cached" for performance (so you don't need to seek and  
> read, sequential buffer access, etc.)
>
> A sometimes better solution is to have the state returned to the  
> caller, and require the caller to pass/use the state later - then  
> you don't need thread locals.
>
> You can accomplish a similar solution by returning a "SessionKey"  
> object, and have the caller pass this later.  You can then have a  
> WeakHashMap of SessionKey,SearchState that the code can use.  When  
> the SessionKey is destroyed (no longer referenced), the state map  
> can be cleaned up automatically.
>
>
>
> On Sep 10, 2008, at 11:30 PM, Noble Paul നോബിള്‍  
> नोब्ळ् wrote:
>
>> When I look at the reference tree That is the feeling I get. if you
>> held a WeakReference it would get released .
>> |- base of org.apache.lucene.index.CompoundFileReader$CSIndexInput
>>              |- input of org.apache.lucene.index.SegmentTermEnum
>>                  |- value of java.lang.ThreadLocal$ThreadLocalMap 
>> $Entry
>>
>> On Wed, Sep 10, 2008 at 8:39 PM, Chris Lu <chris.lu@gmail.com> wrote:
>>> Does this make any difference?
>>> If I intentionally close the searcher and reader failed to release  
>>> the
>>> memory, I can not rely on some magic of JVM to release it.
>>> --
>>> Chris Lu
>>> -------------------------
>>> Instant Scalable Full-Text Search On Any Database/Application
>>> site: http://www.dbsight.net
>>> demo: http://search.dbsight.com
>>> Lucene Database Search in 3 minutes:
>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>> DBSight customer, a shopping comparison site, (anonymous per  
>>> request) got
>>> 2.6 Million Euro funding!
>>>
>>> On Wed, Sep 10, 2008 at 4:03 AM, Noble Paul നോബിള്‍  
>>> नोब्ळ्
>>> <noble.paul@gmail.com> wrote:
>>>>
>>>> Why do you need to keep a strong reference?
>>>> Why not a WeakReference ?
>>>>
>>>> --Noble
>>>>
>>>> On Wed, Sep 10, 2008 at 12:27 AM, Chris Lu <chris.lu@gmail.com>  
>>>> wrote:
>>>>> The problem should be similar to what's talked about on this  
>>>>> discussion.
>>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal
>>>>>
>>>>> There is a memory leak for Lucene search from Lucene-1195.(svn  
>>>>> r659602,
>>>>> May23,2008)
>>>>>
>>>>> This patch brings in a ThreadLocal cache to TermInfosReader.
>>>>>
>>>>> It's usually recommended to keep the reader open, and reuse it  
>>>>> when
>>>>> possible. In a common J2EE application, the http requests are  
>>>>> usually
>>>>> handled by different threads. But since the cache is  
>>>>> ThreadLocal, the
>>>>> cache
>>>>> are not really usable by other threads. What's worse, the cache  
>>>>> can not
>>>>> be
>>>>> cleared by another thread!
>>>>>
>>>>> This leak is not so obvious usually. But my case is using  
>>>>> RAMDirectory,
>>>>> having several hundred megabytes. So one un-released resource is  
>>>>> obvious
>>>>> to
>>>>> me.
>>>>>
>>>>> Here is the reference tree:
>>>>> org.apache.lucene.store.RAMDirectory
>>>>> |- directory of org.apache.lucene.store.RAMFile
>>>>>     |- file of org.apache.lucene.store.RAMInputStream
>>>>>         |- base of
>>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput
>>>>>             |- input of org.apache.lucene.index.SegmentTermEnum
>>>>>                 |- value of java.lang.ThreadLocal$ThreadLocalMap 
>>>>> $Entry
>>>>>
>>>>>
>>>>> After I switched back to svn revision 659601, right before this  
>>>>> patch is
>>>>> checked in, the memory leak is gone.
>>>>> Although my case is RAMDirectory, I believe this will affect  
>>>>> disk based
>>>>> index also.
>>>>>
>>>>> --
>>>>> Chris Lu
>>>>> -------------------------
>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>> site: http://www.dbsight.net
>>>>> demo: http://search.dbsight.com
>>>>> Lucene Database Search in 3 minutes:
>>>>>
>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>> DBSight customer, a shopping comparison site, (anonymous per  
>>>>> request)
>>>>> got
>>>>> 2.6 Million Euro funding!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>
>>>
>>>
>>
>>
>>
>> -- 
>> --Noble Paul
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message