lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Bellew <>
Subject Re: CloseableThreadLocal problem
Date Fri, 02 Mar 2012 23:56:49 GMT
Thanks for your reply MIke, I create this bug.


On Thu, Mar 1, 2012 at 2:32 PM, Michael McCandless
<> wrote:
> Phew, tricky.
> The problem is purging is potentially costly... it iterates all
> entries in the map (threads that have called get) looking for dead
> threads.
> Can you open an issue...?  We can iterate there.  Thanks for raising this,
> Mike McCandless
> On Wed, Feb 29, 2012 at 12:17 PM, Matthew Bellew <> wrote:
>> We tracked down a large memory leak (effectively a leak anyway) caused
>> by how Analyzer users CloseableThreadLocal.
>> CloseableThreadLocal.hardRefs holds references to Thread objects as
>> keys.  The problem is that it only frees these references in the set()
>> method, and SnowballAnalyzer will only call set() when it is used by a
>> NEW thread.
>> The problem scenario is as follows:
>> The server experiences a spike in usage (say by robots or whatever)
>> and many threads are created and referenced by
>> CloseableThreadLocal.hardRefs.  The server quiesces and lets many of
>> these threads expire normally.  Now we have a smaller, but adequate
>> thread pool.  So CloseableThreadLocal.set() may not be called by
>> SnowBallAnalyzer (via Analyzer) for a _long_ time.  The purge code is
>> never called, and these threads along with their thread local storage
>> (lucene related or not) is never cleaned up.
>> I think calling the purge code in both get() and set() would have
>> avoided this problem, perhaps using WeakHashMap instead of HashMap may
>> also have helped (WeakHashMap purges on get() and set())
>> Our current work around is to not share SnowBallAnalyzer instances
>> among HTTP searcher threads.  We open and close one on every request.
>> Thanks,
>> Matt
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message