lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .
Date Thu, 03 Jan 2013 00:58:20 GMT
Hi Alon,

CloseableThreadLocal has logic that tries to purge dead threads
periodically (the countUntilPurge that you described) ... but it's
only "best effort" / heuristic, and, indeed, if lots of threads have
come through, and then suddenly all is quiet and the threads die off,
then CTL will still hold those instances, until new searches arrive
and purge() is eventually called.

But for this to account for 470 MB of heap seems absurdly high ... how
many threads come through Lucene at the peak?

Which Directory impl are you using?

To be sure we have the right issue, can you limit that max thread
count to something very low (eg, 1 thread), run the same test, and
then see if RAM doesn't grow much beyond that initial 30 MB?

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jan 2, 2013 at 7:29 PM, Alon Muchnick <alon@datonics.com> wrote:
> hi MIke,
> thank you very much for replying .
>
> i am not sure when the doClose() method of the SegementReader will be
> called  ,
> but i think you are correct regarding the SegmentReader still being live ,
> we have an singleton object with a hash map of several RamDirectory indexes
> and their IndexReader, IndexSearcher and IndexWriter objects ,
> this object is initialized when the system starts and only closed when the
> system shuts down .
>
> the upgrade to tomcat 7 did increase the amount of live threads (due to
> changes in tomcat's thread pool behavior ) and could explain more RAM being
> used , but the scenario i am seeing is that even after the application had
> stopped receiving new connection ,the thread pool is reduced to minimum
> size and GC had been run , the are still references to the closed threads
> as the keys in the weak hash map.
>
> am i wrong to assume that under the above scenario the weakhash map should
> have been truncated to hold only keys for the remaining live threads in the
> the thread pool ? or will this never happen because there will be no more
> additonal search requests thus the countUntilPurge in CloseableThreadLocal
> will never reach 0 and the purge method will be never called?
>
>
> Just to be clear regarding the sevirity of this , when the system comes up
> the 6 RAMDirectory Lucune indexes are about 30 MG , after a few hours
> ,when we sample it after the thread pool returned to minimum size and a GC
> had been run the Lucune indexes take 500MG .
>
> i guess its not clear to me when will the memory in  CloseableThreadLocals
> object will be release when working with a thread pool  ?
>
>
> Thanks in advanced
>
> Alon
>
>
> On Wed, Jan 2, 2013 at 10:46 PM, Michael McCandless <
> lucene@mikemccandless.com <javascript:_e({}, 'cvml',
> 'lucene@mikemccandless.com');>> wrote:
>
>> CloseableThreadLocal.hardRefs is still a live reference then it
>> means the SegmentReader hasn't been closed yet, ie I think the chain
>> you show below is from a still-open SegmentReader?
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message