lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Thu, 11 Sep 2008 03:12:34 GMT
I attached it to JIRA 1195.

On Sep 10, 2008, at 9:26 PM, Grant Ingersoll wrote:

> If you're intending to donate this, please attach it to a JIRA and  
> check the "Grant to ASF" checkbox (or whatever it's called).   
> Otherwise, we can't use it just from an email.
>
> Thanks,
> Grant
>
> On Sep 10, 2008, at 9:53 PM, robert engels wrote:
>
>> Here is SafeThreadLocal. There is a method purge() that can be  
>> called after a IndexReader is closed. Due to GC, this may not be  
>> sufficient, better code would look like this:
>>
>> 	SomeLargeObject slo; // maybe a RAMDirectory?
>> 	try {
>> 		slo = new SomeLargeObject(); // or other creation mechanism;
>> 	} catch (OutOfMemoryException e) {
>> 		SafeThreadLocal.purge();
>> 		// now try again
>> 		slo = new SomeLargeObject(); // or other creation mechanism;
>> 	}
>>
>> 		slo = new SomeLargeObject(); // or other creation mechanism;
>>
>> Even without the above, it is far more aggressive in releasing the  
>> memory, since any access of the SafeThreadLocal will purge stale  
>> entries (WeakHashMap functionality).
>>
>> SafeThreadLocal.java
>>
>> package org.apache.lucene.util;
>>
>> import java.lang.ref.WeakReference;
>> import java.util.*;
>>
>> /**
>>  * version of ThreadLocal that is more deterministic in memory usage
>>  */
>> public class SafeThreadLocal {
>>     private final static Set locals = new HashSet();
>>     private final Map values = new WeakHashMap();
>>
>>     public SafeThreadLocal() {
>> 	synchronized(SafeThreadLocal.class) {
>> 	    locals.add(new WeakReference(this));
>> 	}
>> 	// maybe call purge() here ? makes things slower, but a bit safer
>>     }
>>
>>     protected Object initialValue() {
>>         return null;
>>     }
>>
>>     public final synchronized Object get() {
>>         final Object key = Thread.currentThread();
>>         Object value;
>>         if(!values.containsKey(key)) {
>>             value = initialValue();
>>             values.put(key,value);
>>         } else {
>>             value = values.get(key);
>>         }
>>         return value;
>>     }
>>
>>     public final synchronized void set(Object value) {
>>         values.put(Thread.currentThread(),value);
>>     }
>>
>>     /**
>>      * clear any stale entries across all thread locals
>>      */
>>     public static synchronized void purge() {
>> 	for(Iterator i = locals.iterator();i.hasNext();) {
>> 	    SafeThreadLocal sfl = (SafeThreadLocal) ((WeakReference)i.next 
>> ()).get();
>> 	    if(sfl==null)
>> 		i.remove();
>> 	    else {
>> 		synchronized(sfl) {
>> 		    sfl.locals.size(); // causes stale entries to be purged
>> 		}
>> 	    }
>> 	}
>>     }
>> }
>>
>> 		
>>
>> On Sep 10, 2008, at 12:43 PM, Chris Lu wrote:
>>
>>> SafeThreadLocal is very interesting. It'll be good not only for  
>>> Lucene, but also other projects.
>>>
>>> Could you please post it?
>>>
>>> -- 
>>> Chris Lu
>>> -------------------------
>>> Instant Scalable Full-Text Search On Any Database/Application
>>> site: http://www.dbsight.net
>>> demo: http://search.dbsight.com
>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>> DBSight customer, a shopping comparison site, (anonymous per  
>>> request) got 2.6 Million Euro funding!
>>>
>>>
>>> On Wed, Sep 10, 2008 at 9:41 AM, robert engels  
>>> <rengels@ix.netcom.com> wrote:
>>> The other thing Lucene can do is create a SafeThreadLocal - it is  
>>> rather trivial, and have that integrate at a higher-level,  
>>> allowing for manual clean-up across all threads.
>>>
>>> It MIGHT  be a bit slower than the JDK version (since that uses  
>>> heuristics to clear stale entries), and so doesn't always clear.
>>>
>>> But it will be far more deterministic.
>>>
>>> If someone is interested I can post the class, but I think it is  
>>> well within the understanding of the core Lucene developers.
>>>
>>>
>>> On Sep 10, 2008, at 11:10 AM, robert engels wrote:
>>>
>>>> You do not need to create a new RAMDirectory - just write to the  
>>>> existing one, and then reopen() the IndexReader using it.
>>>>
>>>> This will prevent lots of big objects being created. This may be  
>>>> the source of your problem.
>>>>
>>>> Even if the Segment is closed, the ThreadLocal will no longer be  
>>>> referenced, but there will still be a reference to the  
>>>> SegmentTermEnum (which will be cleared when the thread dies, or  
>>>> "most likely" when new thread locals on that thread a created,  
>>>> so here is a potential problem.
>>>>
>>>> Thread 1 does a search, creates a thread local that references  
>>>> the RAMDir (A).
>>>> Thread 2 does a search, creates a thread local that references  
>>>> the RAMDir (A).
>>>>
>>>> All readers, are closed on RAMDir (A).
>>>>
>>>> A new RAMDir (B) is opened.
>>>>
>>>> There may still be references in the thread local maps to RAMDir  
>>>> A (since no new thread local have been created yet).
>>>>
>>>> So you may get OOM depending on the size of the RAMDir (since  
>>>> you would need room for more than 1).  If you extend this out  
>>>> with lots of threads that don't run very often, you can see how  
>>>> you could easily run out of memory.  "I think" that ThreadLocal  
>>>> should use a ReferenceQueue so stale object slots can be  
>>>> reclaimed as soon as the key is dereferenced - but that is an  
>>>> issue for SUN.
>>>>
>>>> This is why you don't want to create new RAMDirs.
>>>>
>>>> A good rule of thumb - don't keep references to large objects in  
>>>> ThreadLocal (especially indirectly).  If needed, use a "key",  
>>>> and then read the cache using a the "key".
>>>> This would be something for the Lucene folks to change.
>>>>
>>>> On Sep 10, 2008, at 10:44 AM, Chris Lu wrote:
>>>>
>>>>> I am really want to find out where I am doing wrong, if that's  
>>>>> the case.
>>>>>
>>>>> Yes. I have made certain that I closed all Readers/Searchers,  
>>>>> and verified that through memory profiler.
>>>>>
>>>>> Yes. I am creating new RAMDirectory. But that's the problem. I  
>>>>> need to update the content. Sure, if no content update and  
>>>>> everything the same, of course no OOM.
>>>>>
>>>>> Yes. No guarantee of the thread schedule. But that's the  
>>>>> problem. If Lucene is using ThreadLocal to cache lots of things  
>>>>> by the Thread as the key, and no idea when it'll be released.  
>>>>> Of course ThreadLocal is not Lucene's problem...
>>>>>
>>>>> Chris
>>>>>
>>>>> On Wed, Sep 10, 2008 at 8:34 AM, robert engels  
>>>>> <rengels@ix.netcom.com> wrote:
>>>>> It is basic Java. Threads are not guaranteed to run on any sort  
>>>>> of schedule. If you create lots of large objects in one thread,  
>>>>> releasing them in another, there is a good chance you will get  
>>>>> an OOM (since the releasing thread may not run before the OOM  
>>>>> occurs)...  This is not Lucene specific by any means.
>>>>>
>>>>> It is a misunderstanding on your part about how GC works.
>>>>>
>>>>> I assume you must at some point be creating new RAMDirectories  
>>>>> - otherwise the memory would never really increase, since the  
>>>>> IndexReader/enums/etc are not very large...
>>>>>
>>>>> When you create a new RAMDirectories, you need to BE  
>>>>> CERTAIN !!! that the other IndexReaders/Searchers using the old  
>>>>> RAMDirectory are ALL CLOSED, otherwise their memory will still  
>>>>> be in use, which leads to your OOM...
>>>>>
>>>>>
>>>>> On Sep 10, 2008, at 10:16 AM, Chris Lu wrote:
>>>>>
>>>>>> I do not believe I am making any mistake. Actually I just got  
>>>>>> an email from another user, complaining about the same thing.  
>>>>>> And I am having the same usage pattern.
>>>>>>
>>>>>> After the reader is opened, the RAMDirectory is shared by  
>>>>>> several objects.
>>>>>> There is one instance of RAMDirectory in the memory, and it is  
>>>>>> holding lots of memory, which is expected.
>>>>>>
>>>>>> If I close the reader in the same thread that has opened it,  
>>>>>> the RAMDirectory is gone from the memory.
>>>>>> If I close the reader in other threads, the RAMDirectory is  
>>>>>> left in the memory, referenced along the tree I draw in the  
>>>>>> first email.
>>>>>>
>>>>>> I do not think the usage is wrong. Period.
>>>>>>
>>>>>> -------------------------------------
>>>>>> Hi,
>>>>>>
>>>>>>    i found a forum post from you here [1] where you mention  
>>>>>> that you
>>>>>> have a memory leak using the lucene ram directory. I'd like to  
>>>>>> ask you
>>>>>> if you already have resolved the problem and how you did it or  
>>>>>> maybe
>>>>>> you know where i can read about the solution. We are using
>>>>>> RAMDirectory too and figured out, that over time the memory
>>>>>> consumption raises and raises until the system breaks down but  
>>>>>> only
>>>>>> when we performing much index updates. if we only create the  
>>>>>> index and
>>>>>> don't do nothing except searching it, it work fine.
>>>>>>
>>>>>> maybe you can give me a hint or a link,
>>>>>> greetz,
>>>>>> -------------------------------------
>>>>>>
>>>>>> -- 
>>>>>> Chris Lu
>>>>>> -------------------------
>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>> site: http://www.dbsight.net
>>>>>> demo: http://search.dbsight.com
>>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
>>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>> DBSight customer, a shopping comparison site, (anonymous per  
>>>>>> request) got 2.6 Million Euro funding!
>>>>>>
>>>>>> On Wed, Sep 10, 2008 at 7:12 AM, robert engels  
>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>> Sorry, but I am fairly certain you are mistaken.
>>>>>>
>>>>>> If you only have a single IndexReader, the RAMDirectory will  
>>>>>> be shared in all cases.
>>>>>>
>>>>>> The only memory growth is any buffer space allocated by an  
>>>>>> IndexInput (used in many places and cached).
>>>>>>
>>>>>> Normally the IndexInput created by a RAMDirectory do not have  
>>>>>> any buffer allocated, since the underlying store is already in  
>>>>>> memory.
>>>>>>
>>>>>> You have some other problem in your code...
>>>>>>
>>>>>> On Sep 10, 2008, at 1:10 AM, Chris Lu wrote:
>>>>>>
>>>>>>> Actually, even I only use one IndexReader, some resources are
 
>>>>>>> cached via the ThreadLocal cache, and can not be released  
>>>>>>> unless all threads do the close action.
>>>>>>>
>>>>>>> SegmentTermEnum itself is small, but it holds RAMDirectory  
>>>>>>> along the path, which is big.
>>>>>>>
>>>>>>> -- 
>>>>>>> Chris Lu
>>>>>>> -------------------------
>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>> site: http://www.dbsight.net
>>>>>>> demo: http://search.dbsight.com
>>>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/

>>>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>> DBSight customer, a shopping comparison site, (anonymous per
 
>>>>>>> request) got 2.6 Million Euro funding!
>>>>>>>
>>>>>>> On Tue, Sep 9, 2008 at 10:43 PM, robert engels  
>>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>>> You do not need a pool of IndexReaders...
>>>>>>>
>>>>>>> It does not matter what class it is, what matters is the  
>>>>>>> class that ultimately holds the reference.
>>>>>>>
>>>>>>> If the IndexReader is never closed, the SegmentReader(s) is 

>>>>>>> never closed, so the thread local in TermInfosReader is not 

>>>>>>> cleared (because the thread never dies). So you will get one
 
>>>>>>> SegmentTermEnum, per thread * per segment.
>>>>>>>
>>>>>>> The SegmentTermEnum is not a large object, so even if you had
 
>>>>>>> 100 threads, and 100 segments, for 10k instances, seems hard
 
>>>>>>> to believe that is the source of your memory issue.
>>>>>>>
>>>>>>> The SegmentTermEnum is cached by thread since it needs to  
>>>>>>> enumerate the terms, not having a per thread cache, would  
>>>>>>> lead to lots of random access when multiple threads read the
 
>>>>>>> index - very slow.
>>>>>>>
>>>>>>> You need to keep in mind, what if every thread was executing
 
>>>>>>> a search simultaneously - you would still have 100x100  
>>>>>>> SegmentTermEnum instances anyway !  The only way to prevent 

>>>>>>> that would be to create and destroy the SegmentTermEnum on  
>>>>>>> each call (opening and seeking to the proper spot) - which  
>>>>>>> would be SLOW SLOW SLOW.
>>>>>>>
>>>>>>> On Sep 10, 2008, at 12:19 AM, Chris Lu wrote:
>>>>>>>
>>>>>>>> I have tried to create an IndexReader pool and dynamically
 
>>>>>>>> create searcher. But the memory leak is the same. It's not
 
>>>>>>>> related to the Searcher class specifically, but the  
>>>>>>>> SegmentTermEnum in TermInfosReader.
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Chris Lu
>>>>>>>> -------------------------
>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>>> site: http://www.dbsight.net
>>>>>>>> demo: http://search.dbsight.com
>>>>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/

>>>>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>> DBSight customer, a shopping comparison site, (anonymous
per  
>>>>>>>> request) got 2.6 Million Euro funding!
>>>>>>>>
>>>>>>>> On Tue, Sep 9, 2008 at 10:14 PM, robert engels  
>>>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>>>> A searcher uses an IndexReader - the IndexReader is slow
to  
>>>>>>>> open, not a Searcher. And searchers can share an IndexReader.
>>>>>>>>
>>>>>>>> You want to create a single shared (across all threads/ 
>>>>>>>> users) IndexReader (usually), and create an Searcher as 

>>>>>>>> needed and dispose.  It is VERY CHEAP to create the Searcher.
>>>>>>>>
>>>>>>>> I am fairly certain the javadoc on Searcher is incorrect.
  
>>>>>>>> The warning "For performance reasons it is recommended to
 
>>>>>>>> open only one IndexSearcher and use it for all of your  
>>>>>>>> searches" is not true in the case where an IndexReader is
 
>>>>>>>> passed to the ctor.
>>>>>>>>
>>>>>>>> Any caching should USUALLY be performed at the IndexReader
 
>>>>>>>> level.
>>>>>>>>
>>>>>>>> You are most likely using the "path" ctor, and that is the
 
>>>>>>>> source of your problems, as multiple IndexReader instances
 
>>>>>>>> are being created, and thus the memory use.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sep 9, 2008, at 11:44 PM, Chris Lu wrote:
>>>>>>>>
>>>>>>>>> On J2EE environment, usually there is a searcher pool
with  
>>>>>>>>> several searchers open.
>>>>>>>>> The speed to opening a large index for every user is
not  
>>>>>>>>> acceptable.
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Chris Lu
>>>>>>>>> -------------------------
>>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>>>> site: http://www.dbsight.net
>>>>>>>>> demo: http://search.dbsight.com
>>>>>>>>> Lucene Database Search in 3 minutes: http:// 
>>>>>>>>> wiki.dbsight.com/index.php? 
>>>>>>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>>> DBSight customer, a shopping comparison site, (anonymous
 
>>>>>>>>> per request) got 2.6 Million Euro funding!
>>>>>>>>>
>>>>>>>>> On Tue, Sep 9, 2008 at 9:03 PM, robert engels  
>>>>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>>>>> You need to close the searcher within the thread that
is  
>>>>>>>>> using it, in order to have it cleaned up quickly... usually
 
>>>>>>>>> right after you display the page of results.
>>>>>>>>>
>>>>>>>>> If you are keeping multiple searcher refs across multiple
 
>>>>>>>>> threads for paging/whatever, you have not coded it correctly.
>>>>>>>>>
>>>>>>>>> Imagine 10,000 users - storing a searcher for each one
is  
>>>>>>>>> not going to work...
>>>>>>>>>
>>>>>>>>> On Sep 9, 2008, at 10:21 PM, Chris Lu wrote:
>>>>>>>>>
>>>>>>>>>> Right, in a sense I can not release it from another
 
>>>>>>>>>> thread. But that's the problem.
>>>>>>>>>>
>>>>>>>>>> It's a J2EE environment, all threads are kind of
equal.  
>>>>>>>>>> It's simply not possible to iterate through all threads
to  
>>>>>>>>>> close the searcher, thus releasing the ThreadLocal
cache.
>>>>>>>>>> Unless Lucene is not recommended for J2EE environment,
 
>>>>>>>>>> this has to be fixed.
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Chris Lu
>>>>>>>>>> -------------------------
>>>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>>>>> site: http://www.dbsight.net
>>>>>>>>>> demo: http://search.dbsight.com
>>>>>>>>>> Lucene Database Search in 3 minutes: http:// 
>>>>>>>>>> wiki.dbsight.com/index.php? 
>>>>>>>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>>>> DBSight customer, a shopping comparison site, (anonymous
 
>>>>>>>>>> per request) got 2.6 Million Euro funding!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 9, 2008 at 8:14 PM, robert engels  
>>>>>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>>>>>> Your code is not correct. You cannot release it on
another  
>>>>>>>>>> thread - the first thread may creating hundreds/thousands
 
>>>>>>>>>> of instances before the other thread ever runs...
>>>>>>>>>>
>>>>>>>>>> On Sep 9, 2008, at 10:10 PM, Chris Lu wrote:
>>>>>>>>>>
>>>>>>>>>>> If I release it on the thread that's creating
the  
>>>>>>>>>>> searcher, by setting searcher=null, everything
is fine,  
>>>>>>>>>>> the memory is released very cleanly.
>>>>>>>>>>> My load test was to repeatedly create a searcher
on a  
>>>>>>>>>>> RAMDirectory and release it on another thread.
The test  
>>>>>>>>>>> will quickly go to OOM after several runs. I
set the heap  
>>>>>>>>>>> size to be 1024M, and the RAMDirectory is of
size 250M.  
>>>>>>>>>>> Using some profiling tool, the used size simply
stepped  
>>>>>>>>>>> up pretty obviously by 250M.
>>>>>>>>>>>
>>>>>>>>>>> I think we should not rely on something that's
a "maybe"  
>>>>>>>>>>> behavior, especially for a general purpose library.
>>>>>>>>>>>
>>>>>>>>>>> Since it's a multi-threaded env, the thread that's
 
>>>>>>>>>>> creating the entries in the LRU cache may not
go away  
>>>>>>>>>>> quickly(actually most, if not all, application
servers  
>>>>>>>>>>> will try to reuse threads), so the LRU cache,
which uses  
>>>>>>>>>>> thread as the key, can not be released, so the
 
>>>>>>>>>>> SegmentTermEnum which is in the same class can
not be  
>>>>>>>>>>> released.
>>>>>>>>>>>
>>>>>>>>>>> And yes, I close the RAMDirectory, and the fileMap
is  
>>>>>>>>>>> released. I verified that through the profiler
by  
>>>>>>>>>>> directly checking the values in the snapshot.
>>>>>>>>>>>
>>>>>>>>>>> Pretty sure the reference tree wasn't like this
using  
>>>>>>>>>>> code before this commit, because after close
the searcher  
>>>>>>>>>>> in another thread, the RAMDirectory totally disappeared
 
>>>>>>>>>>> from the memory snapshot.
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Chris Lu
>>>>>>>>>>> -------------------------
>>>>>>>>>>> Instant Scalable Full-Text Search On Any Database/

>>>>>>>>>>> Application
>>>>>>>>>>> site: http://www.dbsight.net
>>>>>>>>>>> demo: http://search.dbsight.com
>>>>>>>>>>> Lucene Database Search in 3 minutes: http://

>>>>>>>>>>> wiki.dbsight.com/index.php? 
>>>>>>>>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>>>>> DBSight customer, a shopping comparison site,
(anonymous  
>>>>>>>>>>> per request) got 2.6 Million Euro funding!
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 9, 2008 at 5:03 PM, Michael McCandless
 
>>>>>>>>>>> <lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Chris Lu wrote:
>>>>>>>>>>>
>>>>>>>>>>> The problem should be similar to what's talked
about on  
>>>>>>>>>>> this discussion.
>>>>>>>>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?

>>>>>>>>>>> q=ThreadLocal
>>>>>>>>>>>
>>>>>>>>>>> The "rough" conclusion of that thread is that,
 
>>>>>>>>>>> technically, this isn't a memory leak but rather
a  
>>>>>>>>>>> "delayed freeing" problem.  Ie, it may take longer,
 
>>>>>>>>>>> possibly much longer, than you want for the memory
to be  
>>>>>>>>>>> freed.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There is a memory leak for Lucene search from
Lucene-1195. 
>>>>>>>>>>> (svn r659602, May23,2008)
>>>>>>>>>>>
>>>>>>>>>>> This patch brings in a ThreadLocal cache to TermInfosReader.
>>>>>>>>>>>
>>>>>>>>>>> One thing that confuses me: TermInfosReader was
already  
>>>>>>>>>>> using a ThreadLocal to cache the SegmentTermEnum
 
>>>>>>>>>>> instance.  What was added in this commit (for
 
>>>>>>>>>>> LUCENE-1195) was an LRU cache storing Term ->
TermInfo  
>>>>>>>>>>> instances.  But it seems like it's the SegmentTermEnum
 
>>>>>>>>>>> instance that you're tracing below.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It's usually recommended to keep the reader open,
and  
>>>>>>>>>>> reuse it when
>>>>>>>>>>> possible. In a common J2EE application, the http
requests  
>>>>>>>>>>> are usually
>>>>>>>>>>> handled by different threads. But since the cache
is  
>>>>>>>>>>> ThreadLocal, the cache
>>>>>>>>>>> are not really usable by other threads. What's
worse, the  
>>>>>>>>>>> cache can not be
>>>>>>>>>>> cleared by another thread!
>>>>>>>>>>>
>>>>>>>>>>> This leak is not so obvious usually. But my case
is using  
>>>>>>>>>>> RAMDirectory,
>>>>>>>>>>> having several hundred megabytes. So one un-released
 
>>>>>>>>>>> resource is obvious to
>>>>>>>>>>> me.
>>>>>>>>>>>
>>>>>>>>>>> Here is the reference tree:
>>>>>>>>>>> org.apache.lucene.store.RAMDirectory
>>>>>>>>>>>  |- directory of org.apache.lucene.store.RAMFile
>>>>>>>>>>>     |- file of org.apache.lucene.store.RAMInputStream
>>>>>>>>>>>         |- base of  
>>>>>>>>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput
>>>>>>>>>>>             |- input of  
>>>>>>>>>>> org.apache.lucene.index.SegmentTermEnum
>>>>>>>>>>>                 |- value of java.lang.ThreadLocal

>>>>>>>>>>> $ThreadLocalMap$Entry
>>>>>>>>>>>
>>>>>>>>>>> So you have a RAMDir that has several hundred
MB stored  
>>>>>>>>>>> in it, that you're done with yet through this
path Lucene  
>>>>>>>>>>> is keeping it alive?
>>>>>>>>>>>
>>>>>>>>>>> Did you close the RAMDir?  (which will null its
fileMap  
>>>>>>>>>>> and should also free your memory).
>>>>>>>>>>>
>>>>>>>>>>> Also, that reference tree doesn't show the  
>>>>>>>>>>> ThreadResources class that was added in that
commit --  
>>>>>>>>>>> are you sure this reference tree wasn't before
the commit?
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> ------------------------------------------------------------

>>>>>>>>>>> ---------
>>>>>>>>>>> To unsubscribe, e-mail: java-dev- 
>>>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>>>>> For additional commands, e-mail: java-dev- 
>>>>>>>>>>> help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Chris Lu
>>>>>>>>>>> -------------------------
>>>>>>>>>>> Instant Scalable Full-Text Search On Any Database/

>>>>>>>>>>> Application
>>>>>>>>>>> site: http://www.dbsight.net
>>>>>>>>>>> demo: http://search.dbsight.com
>>>>>>>>>>> Lucene Database Search in 3 minutes: http://

>>>>>>>>>>> wiki.dbsight.com/index.php? 
>>>>>>>>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>>>>> DBSight customer, a shopping comparison site,
(anonymous  
>>>>>>>>>>> per request) got 2.6 Million Euro funding!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>


Mime
View raw message