lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Wed, 10 Sep 2008 16:05:51 GMT

Good question.

As far as I can tell, nowhere in Lucene do we put a SegmentTermEnum  
directly into ThreadLocal, after rev 659602.

Is it possible that output came from a run with Lucene before rev  
659602?

Mike

Chris Lu wrote:

> Is it possible that some other places that's using SegmentTermEnum  
> as ThreadLocal?
> This may explain why TermInfosReader.ThreadResources is not in the  
> memory snapshot.
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
> On Wed, Sep 10, 2008 at 2:45 AM, Michael McCandless <lucene@mikemccandless.com 
> > wrote:
>
> I still don't quite understand what's causing your memory growth.
>
> SegmentTermEnum insances have been held in a ThreadLocal cache in  
> TermInfosReader for a very long time (at least since Lucene 1.4).
>
> If indeed it's the RAMDir's contents being kept "alive" due to this,  
> then, you should have already been seeing this problem before rev  
> 659602.  And I still don't get why your reference tree is missing  
> the TermInfosReader.ThreadResources class.
>
> I'd like to understand the root cause before we hash out possible  
> solutions.
>
> Can you post the sources for your load test?
>
> Mike
>
>
> Chris Lu wrote:
>
> Actually, even I only use one IndexReader, some resources are cached  
> via the ThreadLocal cache, and can not be released unless all  
> threads do the close action.
>
> SegmentTermEnum itself is small, but it holds RAMDirectory along the  
> path, which is big.
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
> On Tue, Sep 9, 2008 at 10:43 PM, robert engels  
> <rengels@ix.netcom.com> wrote:
> You do not need a pool of IndexReaders...
>
> It does not matter what class it is, what matters is the class that  
> ultimately holds the reference.
>
> If the IndexReader is never closed, the SegmentReader(s) is never  
> closed, so the thread local in TermInfosReader is not cleared  
> (because the thread never dies). So you will get one  
> SegmentTermEnum, per thread * per segment.
>
> The SegmentTermEnum is not a large object, so even if you had 100  
> threads, and 100 segments, for 10k instances, seems hard to believe  
> that is the source of your memory issue.
>
> The SegmentTermEnum is cached by thread since it needs to enumerate  
> the terms, not having a per thread cache, would lead to lots of  
> random access when multiple threads read the index - very slow.
>
> You need to keep in mind, what if every thread was executing a  
> search simultaneously - you would still have 100x100 SegmentTermEnum  
> instances anyway !  The only way to prevent that would be to create  
> and destroy the SegmentTermEnum on each call (opening and seeking to  
> the proper spot) - which would be SLOW SLOW SLOW.
>
> On Sep 10, 2008, at 12:19 AM, Chris Lu wrote:
>
> I have tried to create an IndexReader pool and dynamically create  
> searcher. But the memory leak is the same. It's not related to the  
> Searcher class specifically, but the SegmentTermEnum in  
> TermInfosReader.
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
> On Tue, Sep 9, 2008 at 10:14 PM, robert engels  
> <rengels@ix.netcom.com> wrote:
> A searcher uses an IndexReader - the IndexReader is slow to open,  
> not a Searcher. And searchers can share an IndexReader.
>
> You want to create a single shared (across all threads/users)  
> IndexReader (usually), and create an Searcher as needed and  
> dispose.  It is VERY CHEAP to create the Searcher.
>
> I am fairly certain the javadoc on Searcher is incorrect.  The  
> warning "For performance reasons it is recommended to open only one  
> IndexSearcher and use it for all of your searches" is not true in  
> the case where an IndexReader is passed to the ctor.
>
> Any caching should USUALLY be performed at the IndexReader level.
>
> You are most likely using the "path" ctor, and that is the source of  
> your problems, as multiple IndexReader instances are being created,  
> and thus the memory use.
>
>
> On Sep 9, 2008, at 11:44 PM, Chris Lu wrote:
>
> On J2EE environment, usually there is a searcher pool with several  
> searchers open.
> The speed to opening a large index for every user is not acceptable.
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
> On Tue, Sep 9, 2008 at 9:03 PM, robert engels  
> <rengels@ix.netcom.com> wrote:
> You need to close the searcher within the thread that is using it,  
> in order to have it cleaned up quickly... usually right after you  
> display the page of results.
>
> If you are keeping multiple searcher refs across multiple threads  
> for paging/whatever, you have not coded it correctly.
>
> Imagine 10,000 users - storing a searcher for each one is not going  
> to work...
>
> On Sep 9, 2008, at 10:21 PM, Chris Lu wrote:
>
> Right, in a sense I can not release it from another thread. But  
> that's the problem.
>
> It's a J2EE environment, all threads are kind of equal. It's simply  
> not possible to iterate through all threads to close the searcher,  
> thus releasing the ThreadLocal cache.
> Unless Lucene is not recommended for J2EE environment, this has to  
> be fixed.
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
>
> On Tue, Sep 9, 2008 at 8:14 PM, robert engels  
> <rengels@ix.netcom.com> wrote:
> Your code is not correct. You cannot release it on another thread -  
> the first thread may creating hundreds/thousands of instances before  
> the other thread ever runs...
>
> On Sep 9, 2008, at 10:10 PM, Chris Lu wrote:
>
> If I release it on the thread that's creating the searcher, by  
> setting searcher=null, everything is fine, the memory is released  
> very cleanly.
> My load test was to repeatedly create a searcher on a RAMDirectory  
> and release it on another thread. The test will quickly go to OOM  
> after several runs. I set the heap size to be 1024M, and the  
> RAMDirectory is of size 250M. Using some profiling tool, the used  
> size simply stepped up pretty obviously by 250M.
>
> I think we should not rely on something that's a "maybe" behavior,  
> especially for a general purpose library.
>
> Since it's a multi-threaded env, the thread that's creating the  
> entries in the LRU cache may not go away quickly(actually most, if  
> not all, application servers will try to reuse threads), so the LRU  
> cache, which uses thread as the key, can not be released, so the  
> SegmentTermEnum which is in the same class can not be released.
>
> And yes, I close the RAMDirectory, and the fileMap is released. I  
> verified that through the profiler by directly checking the values  
> in the snapshot.
>
> Pretty sure the reference tree wasn't like this using code before  
> this commit, because after close the searcher in another thread, the  
> RAMDirectory totally disappeared from the memory snapshot.
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
> On Tue, Sep 9, 2008 at 5:03 PM, Michael McCandless <lucene@mikemccandless.com 
> > wrote:
>
> Chris Lu wrote:
>
> The problem should be similar to what's talked about on this  
> discussion.
> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal
>
> The "rough" conclusion of that thread is that, technically, this  
> isn't a memory leak but rather a "delayed freeing" problem.  Ie, it  
> may take longer, possibly much longer, than you want for the memory  
> to be freed.
>
>
> There is a memory leak for Lucene search from Lucene-1195.(svn  
> r659602, May23,2008)
>
> This patch brings in a ThreadLocal cache to TermInfosReader.
>
> One thing that confuses me: TermInfosReader was already using a  
> ThreadLocal to cache the SegmentTermEnum instance.  What was added  
> in this commit (for LUCENE-1195) was an LRU cache storing Term ->  
> TermInfo instances.  But it seems like it's the SegmentTermEnum  
> instance that you're tracing below.
>
>
> It's usually recommended to keep the reader open, and reuse it when
> possible. In a common J2EE application, the http requests are usually
> handled by different threads. But since the cache is ThreadLocal,  
> the cache
> are not really usable by other threads. What's worse, the cache can  
> not be
> cleared by another thread!
>
> This leak is not so obvious usually. But my case is using  
> RAMDirectory,
> having several hundred megabytes. So one un-released resource is  
> obvious to
> me.
>
> Here is the reference tree:
> org.apache.lucene.store.RAMDirectory
>  |- directory of org.apache.lucene.store.RAMFile
>    |- file of org.apache.lucene.store.RAMInputStream
>        |- base of org.apache.lucene.index.CompoundFileReader 
> $CSIndexInput
>            |- input of org.apache.lucene.index.SegmentTermEnum
>                |- value of java.lang.ThreadLocal$ThreadLocalMap$Entry
>
> So you have a RAMDir that has several hundred MB stored in it, that  
> you're done with yet through this path Lucene is keeping it alive?
>
> Did you close the RAMDir?  (which will null its fileMap and should  
> also free your memory).
>
> Also, that reference tree doesn't show the ThreadResources class  
> that was added in that commit -- are you sure this reference tree  
> wasn't before the commit?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message