lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Bellew <matth...@labkey.com>
Subject Re: CloseableThreadLocal problem
Date Fri, 02 Mar 2012 23:56:49 GMT
Thanks for your reply MIke, I create this bug.

https://issues.apache.org/jira/browse/LUCENE-3841

Matt

On Thu, Mar 1, 2012 at 2:32 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Phew, tricky.
>
> The problem is purging is potentially costly... it iterates all
> entries in the map (threads that have called get) looking for dead
> threads.
>
> Can you open an issue...?  We can iterate there.  Thanks for raising this,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Feb 29, 2012 at 12:17 PM, Matthew Bellew <matthewb@labkey.com> wrote:
>> We tracked down a large memory leak (effectively a leak anyway) caused
>> by how Analyzer users CloseableThreadLocal.
>> CloseableThreadLocal.hardRefs holds references to Thread objects as
>> keys.  The problem is that it only frees these references in the set()
>> method, and SnowballAnalyzer will only call set() when it is used by a
>> NEW thread.
>>
>> The problem scenario is as follows:
>>
>> The server experiences a spike in usage (say by robots or whatever)
>> and many threads are created and referenced by
>> CloseableThreadLocal.hardRefs.  The server quiesces and lets many of
>> these threads expire normally.  Now we have a smaller, but adequate
>> thread pool.  So CloseableThreadLocal.set() may not be called by
>> SnowBallAnalyzer (via Analyzer) for a _long_ time.  The purge code is
>> never called, and these threads along with their thread local storage
>> (lucene related or not) is never cleaned up.
>>
>> I think calling the purge code in both get() and set() would have
>> avoided this problem, perhaps using WeakHashMap instead of HashMap may
>> also have helped (WeakHashMap purges on get() and set())
>>
>> Our current work around is to not share SnowBallAnalyzer instances
>> among HTTP searcher threads.  We open and close one on every request.
>>
>> Thanks,
>> Matt
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message