lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Lu" <chris...@gmail.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Wed, 10 Sep 2008 17:43:11 GMT
SafeThreadLocal is very interesting. It'll be good not only for Lucene, but
also other projects.

Could you please post it?

-- 
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer, a shopping comparison site, (anonymous per request) got
2.6 Million Euro funding!

On Wed, Sep 10, 2008 at 9:41 AM, robert engels <rengels@ix.netcom.com>wrote:

> The other thing Lucene can do is create a SafeThreadLocal - it is rather
> trivial, and have that integrate at a higher-level, allowing for manual
> clean-up across all threads.
> It MIGHT  be a bit slower than the JDK version (since that uses heuristics
> to clear stale entries), and so doesn't always clear.
>
> But it will be far more deterministic.
>
> If someone is interested I can post the class, but I think it is well
> within the understanding of the core Lucene developers.
>
>
> On Sep 10, 2008, at 11:10 AM, robert engels wrote:
>
> You do not need to create a new RAMDirectory - just write to the existing
> one, and then reopen() the IndexReader using it.
> This will prevent lots of big objects being created. This may be the source
> of your problem.
>
> Even if the Segment is closed, the ThreadLocal will no longer be
> referenced, but there will still be a reference to the SegmentTermEnum
> (which will be cleared when the thread dies, or "most likely" when new
> thread locals on that thread a created, so here is a potential problem.
>
> Thread 1 does a search, creates a thread local that references the RAMDir
> (A).
> Thread 2 does a search, creates a thread local that references the RAMDir
> (A).
>
> All readers, are closed on RAMDir (A).
>
> A new RAMDir (B) is opened.
>
> There may still be references in the thread local maps to RAMDir A (since
> no new thread local have been created yet).
>
> So you may get OOM depending on the size of the RAMDir (since you would
> need room for more than 1).  If you extend this out with lots of threads
> that don't run very often, you can see how you could easily run out of
> memory.  "I think" that ThreadLocal should use a ReferenceQueue so stale
> object slots can be reclaimed as soon as the key is dereferenced - but that
> is an issue for SUN.
>
> This is why you don't want to create new RAMDirs.
>
> A good rule of thumb - don't keep references to large objects in
> ThreadLocal (especially indirectly).  If needed, use a "key", and then read
> the cache using a the "key".
> This would be something for the Lucene folks to change.
>
> On Sep 10, 2008, at 10:44 AM, Chris Lu wrote:
>
> I am really want to find out where I am doing wrong, if that's the case.
>
> Yes. I have made certain that I closed all Readers/Searchers, and verified
> that through memory profiler.
> Yes. I am creating new RAMDirectory. But that's the problem. I need to
> update the content. Sure, if no content update and everything the same, of
> course no OOM.
>
> Yes. No guarantee of the thread schedule. But that's the problem. If Lucene
> is using ThreadLocal to cache lots of things by the Thread as the key, and
> no idea when it'll be released. Of course ThreadLocal is not Lucene's
> problem...
>
> Chris
>
> On Wed, Sep 10, 2008 at 8:34 AM, robert engels <rengels@ix.netcom.com>wrote:
>
>>  It is basic Java. Threads are not guaranteed to run on any sort of
>> schedule. If you create lots of large objects in one thread, releasing them
>> in another, there is a good chance you will get an OOM (since the releasing
>> thread may not run before the OOM occurs)...  This is not Lucene specific by
>> any means.
>> It is a misunderstanding on your part about how GC works.
>>
>> I assume you must at some point be creating new RAMDirectories - otherwise
>> the memory would never really increase, since the IndexReader/enums/etc are
>> not very large...
>>
>> When you create a new RAMDirectories, you need to BE CERTAIN !!! that the
>> other IndexReaders/Searchers using the old RAMDirectory are ALL CLOSED,
>> otherwise their memory will still be in use, which leads to your OOM...
>>
>>
>> On Sep 10, 2008, at 10:16 AM, Chris Lu wrote:
>>
>> I do not believe I am making any mistake. Actually I just got an email
>> from another user, complaining about the same thing. And I am having the
>> same usage pattern.
>> After the reader is opened, the RAMDirectory is shared by several objects.
>> There is one instance of RAMDirectory in the memory, and it is holding
>> lots of memory, which is expected.
>>
>> If I close the reader in the same thread that has opened it, the
>> RAMDirectory is gone from the memory.
>> If I close the reader in other threads, the RAMDirectory is left in the
>> memory, referenced along the tree I draw in the first email.
>>
>> I do not think the usage is wrong. Period.
>>
>> -------------------------------------
>>
>> Hi,
>>
>>    i found a forum post from you here [1] where you mention that you
>> have a memory leak using the lucene ram directory. I'd like to ask you
>> if you already have resolved the problem and how you did it or maybe
>> you know where i can read about the solution. We are using
>> RAMDirectory too and figured out, that over time the memory
>> consumption raises and raises until the system breaks down but only
>> when we performing much index updates. if we only create the index and
>> don't do nothing except searching it, it work fine.
>>
>> maybe you can give me a hint or a link,
>> greetz,
>>
>> -------------------------------------
>>
>> --
>> Chris Lu
>> -------------------------
>> Instant Scalable Full-Text Search On Any Database/Application
>> site: http://www.dbsight.net
>> demo: http://search.dbsight.com
>> Lucene Database Search in 3 minutes:
>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>> DBSight customer, a shopping comparison site, (anonymous per request) got
>> 2.6 Million Euro funding!
>>
>> On Wed, Sep 10, 2008 at 7:12 AM, robert engels <rengels@ix.netcom.com>wrote:
>>
>>> Sorry, but I am fairly certain you are mistaken.
>>> If you only have a single IndexReader, the RAMDirectory will be shared in
>>> all cases.
>>>
>>> The only memory growth is any buffer space allocated by an IndexInput
>>> (used in many places and cached).
>>>
>>> Normally the IndexInput created by a RAMDirectory do not have any buffer
>>> allocated, since the underlying store is already in memory.
>>>
>>> You have some other problem in your code...
>>>
>>> On Sep 10, 2008, at 1:10 AM, Chris Lu wrote:
>>>
>>> Actually, even I only use one IndexReader, some resources are cached via
>>> the ThreadLocal cache, and can not be released unless all threads do the
>>> close action.
>>>
>>> SegmentTermEnum itself is small, but it holds RAMDirectory along the
>>> path, which is big.
>>>
>>> --
>>> Chris Lu
>>> -------------------------
>>> Instant Scalable Full-Text Search On Any Database/Application
>>> site: http://www.dbsight.net
>>> demo: http://search.dbsight.com
>>> Lucene Database Search in 3 minutes:
>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>> DBSight customer, a shopping comparison site, (anonymous per request) got
>>> 2.6 Million Euro funding!
>>> On Tue, Sep 9, 2008 at 10:43 PM, robert engels <rengels@ix.netcom.com>wrote:
>>>
>>>>  You do not need a pool of IndexReaders...
>>>> It does not matter what class it is, what matters is the class that
>>>> ultimately holds the reference.
>>>>
>>>> If the IndexReader is never closed, the SegmentReader(s) is never
>>>> closed, so the thread local in TermInfosReader is not cleared (because the
>>>> thread never dies). So you will get one SegmentTermEnum, per thread * per
>>>> segment.
>>>>
>>>> The SegmentTermEnum is not a large object, so even if you had 100
>>>> threads, and 100 segments, for 10k instances, seems hard to believe that
is
>>>> the source of your memory issue.
>>>>
>>>> The SegmentTermEnum is cached by thread since it needs to enumerate the
>>>> terms, not having a per thread cache, would lead to lots of random access
>>>> when multiple threads read the index - very slow.
>>>>
>>>> You need to keep in mind, what if every thread was executing a search
>>>> simultaneously - you would still have 100x100 SegmentTermEnum instances
>>>> anyway !  The only way to prevent that would be to create and destroy the
>>>> SegmentTermEnum on each call (opening and seeking to the proper spot) -
>>>> which would be SLOW SLOW SLOW.
>>>>
>>>> On Sep 10, 2008, at 12:19 AM, Chris Lu wrote:
>>>>
>>>> I have tried to create an IndexReader pool and dynamically create
>>>> searcher. But the memory leak is the same. It's not related to the Searcher
>>>> class specifically, but the SegmentTermEnum in TermInfosReader.
>>>>
>>>> --
>>>> Chris Lu
>>>> -------------------------
>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>> site: http://www.dbsight.net
>>>> demo: http://search.dbsight.com
>>>> Lucene Database Search in 3 minutes:
>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>> DBSight customer, a shopping comparison site, (anonymous per request)
>>>> got 2.6 Million Euro funding!
>>>>
>>>> On Tue, Sep 9, 2008 at 10:14 PM, robert engels <rengels@ix.netcom.com>wrote:
>>>>
>>>>>  A searcher uses an IndexReader - the IndexReader is slow to open, not
>>>>> a Searcher. And searchers can share an IndexReader.
>>>>> You want to create a single shared (across all threads/users)
>>>>> IndexReader (usually), and create an Searcher as needed and dispose.
 It is
>>>>> VERY CHEAP to create the Searcher.
>>>>>
>>>>> I am fairly certain the javadoc on Searcher is incorrect.  The warning
>>>>> "For performance reasons it is recommended to open only one
>>>>> IndexSearcher and use it for all of your searches" is not true in the
>>>>> case where an IndexReader is passed to the ctor.
>>>>>
>>>>> Any caching should USUALLY be performed at the IndexReader level.
>>>>>
>>>>> You are most likely using the "path" ctor, and that is the source of
>>>>> your problems, as multiple IndexReader instances are being created, and
thus
>>>>> the memory use.
>>>>>
>>>>>
>>>>> On Sep 9, 2008, at 11:44 PM, Chris Lu wrote:
>>>>>
>>>>> On J2EE environment, usually there is a searcher pool with several
>>>>> searchers open. The speed to opening a large index for every user is
>>>>> not acceptable.
>>>>>
>>>>> --
>>>>> Chris Lu
>>>>> -------------------------
>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>> site: http://www.dbsight.net
>>>>> demo: http://search.dbsight.com
>>>>> Lucene Database Search in 3 minutes:
>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>> DBSight customer, a shopping comparison site, (anonymous per request)
>>>>> got 2.6 Million Euro funding!
>>>>>
>>>>> On Tue, Sep 9, 2008 at 9:03 PM, robert engels <rengels@ix.netcom.com>wrote:
>>>>>
>>>>>> You need to close the searcher within the thread that is using it,
in
>>>>>> order to have it cleaned up quickly... usually right after you display
the
>>>>>> page of results.
>>>>>> If you are keeping multiple searcher refs across multiple threads
for
>>>>>> paging/whatever, you have not coded it correctly.
>>>>>>
>>>>>> Imagine 10,000 users - storing a searcher for each one is not going
to
>>>>>> work...
>>>>>>
>>>>>> On Sep 9, 2008, at 10:21 PM, Chris Lu wrote:
>>>>>>
>>>>>> Right, in a sense I can not release it from another thread. But that's
>>>>>> the problem.
>>>>>>
>>>>>> It's a J2EE environment, all threads are kind of equal. It's simply
>>>>>> not possible to iterate through all threads to close the searcher,
thus
>>>>>> releasing the ThreadLocal cache.
>>>>>> Unless Lucene is not recommended for J2EE environment, this has to
be
>>>>>> fixed.
>>>>>>
>>>>>> --
>>>>>> Chris Lu
>>>>>> -------------------------
>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>> site: http://www.dbsight.net
>>>>>> demo: http://search.dbsight.com
>>>>>> Lucene Database Search in 3 minutes:
>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>> DBSight customer, a shopping comparison site, (anonymous per request)
>>>>>> got 2.6 Million Euro funding!
>>>>>>
>>>>>> On Tue, Sep 9, 2008 at 8:14 PM, robert engels <rengels@ix.netcom.com>wrote:
>>>>>>
>>>>>>> Your code is not correct. You cannot release it on another thread
-
>>>>>>> the first thread may creating hundreds/thousands of instances
before the
>>>>>>> other thread ever runs...
>>>>>>>
>>>>>>> On Sep 9, 2008, at 10:10 PM, Chris Lu wrote:
>>>>>>>
>>>>>>> If I release it on the thread that's creating the searcher, by
>>>>>>> setting searcher=null, everything is fine, the memory is released
very
>>>>>>> cleanly.
>>>>>>> My load test was to repeatedly create a searcher on a RAMDirectory
>>>>>>> and release it on another thread. The test will quickly go to
OOM after
>>>>>>> several runs. I set the heap size to be 1024M, and the RAMDirectory
is of
>>>>>>> size 250M. Using some profiling tool, the used size simply stepped
up pretty
>>>>>>> obviously by 250M.
>>>>>>>
>>>>>>> I think we should not rely on something that's a "maybe" behavior,
>>>>>>> especially for a general purpose library.
>>>>>>>
>>>>>>> Since it's a multi-threaded env, the thread that's creating the
>>>>>>> entries in the LRU cache may not go away quickly(actually most,
if not all,
>>>>>>> application servers will try to reuse threads), so the LRU cache,
which uses
>>>>>>> thread as the key, can not be released, so the SegmentTermEnum
which is in
>>>>>>> the same class can not be released.
>>>>>>>
>>>>>>> And yes, I close the RAMDirectory, and the fileMap is released.
I
>>>>>>> verified that through the profiler by directly checking the values
in the
>>>>>>> snapshot.
>>>>>>>
>>>>>>> Pretty sure the reference tree wasn't like this using code before
>>>>>>> this commit, because after close the searcher in another thread,
the
>>>>>>> RAMDirectory totally disappeared from the memory snapshot.
>>>>>>>
>>>>>>> --
>>>>>>> Chris Lu
>>>>>>> -------------------------
>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>> site: http://www.dbsight.net
>>>>>>> demo: http://search.dbsight.com
>>>>>>> Lucene Database Search in 3 minutes:
>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>> DBSight customer, a shopping comparison site, (anonymous per
request)
>>>>>>> got 2.6 Million Euro funding!
>>>>>>>
>>>>>>> On Tue, Sep 9, 2008 at 5:03 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Chris Lu wrote:
>>>>>>>>
>>>>>>>>  The problem should be similar to what's talked about on
this
>>>>>>>>> discussion.
>>>>>>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?q=ThreadLocal
>>>>>>>>>
>>>>>>>>
>>>>>>>> The "rough" conclusion of that thread is that, technically,
this
>>>>>>>> isn't a memory leak but rather a "delayed freeing" problem.
 Ie, it may take
>>>>>>>> longer, possibly much longer, than you want for the memory
to be freed.
>>>>>>>>
>>>>>>>>  There is a memory leak for Lucene search from Lucene-1195.(svn
>>>>>>>>> r659602, May23,2008)
>>>>>>>>>
>>>>>>>>> This patch brings in a ThreadLocal cache to TermInfosReader.
>>>>>>>>>
>>>>>>>>
>>>>>>>> One thing that confuses me: TermInfosReader was already using
a
>>>>>>>> ThreadLocal to cache the SegmentTermEnum instance.  What
was added in this
>>>>>>>> commit (for LUCENE-1195) was an LRU cache storing Term ->
TermInfo
>>>>>>>> instances.  But it seems like it's the SegmentTermEnum instance
that you're
>>>>>>>> tracing below.
>>>>>>>>
>>>>>>>>  It's usually recommended to keep the reader open, and reuse
it when
>>>>>>>>> possible. In a common J2EE application, the http requests
are
>>>>>>>>> usually
>>>>>>>>> handled by different threads. But since the cache is
ThreadLocal,
>>>>>>>>> the cache
>>>>>>>>> are not really usable by other threads. What's worse,
the cache can
>>>>>>>>> not be
>>>>>>>>> cleared by another thread!
>>>>>>>>>
>>>>>>>>> This leak is not so obvious usually. But my case is using
>>>>>>>>> RAMDirectory,
>>>>>>>>> having several hundred megabytes. So one un-released
resource is
>>>>>>>>> obvious to
>>>>>>>>> me.
>>>>>>>>>
>>>>>>>>> Here is the reference tree:
>>>>>>>>> org.apache.lucene.store.RAMDirectory
>>>>>>>>>  |- directory of org.apache.lucene.store.RAMFile
>>>>>>>>>     |- file of org.apache.lucene.store.RAMInputStream
>>>>>>>>>         |- base of
>>>>>>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput
>>>>>>>>>             |- input of org.apache.lucene.index.SegmentTermEnum
>>>>>>>>>                 |- value of
>>>>>>>>> java.lang.ThreadLocal$ThreadLocalMap$Entry
>>>>>>>>>
>>>>>>>>
>>>>>>>> So you have a RAMDir that has several hundred MB stored in
it, that
>>>>>>>> you're done with yet through this path Lucene is keeping
it alive?
>>>>>>>>
>>>>>>>> Did you close the RAMDir?  (which will null its fileMap and
should
>>>>>>>> also free your memory).
>>>>>>>>
>>>>>>>> Also, that reference tree doesn't show the ThreadResources
class
>>>>>>>> that was added in that commit -- are you sure this reference
tree wasn't
>>>>>>>> before the commit?
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chris Lu
>>>>>>> -------------------------
>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>> site: http://www.dbsight.net
>>>>>>> demo: http://search.dbsight.com
>>>>>>> Lucene Database Search in 3 minutes:
>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>> DBSight customer, a shopping comparison site, (anonymous per
request)
>>>>>>> got 2.6 Million Euro funding!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
> --
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes:
> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per request) got
> 2.6 Million Euro funding!
>
>
>
>

Mime
View raw message