Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of matthewb@labkey.com
 designates 209.85.160.48 as permitted sender)
Received-SPF: pass (google.com: domain of matthewb@labkey.com designates
 10.68.229.67 as permitted sender) client-ip=10.68.229.67;
MIME-Version: 1.0
In-Reply-To: 
 <CAL8PwkbENpC5yOqSqqeaS6JMQNBYWtcvdHXn7PKv=ScZdwDLAQ@mail.gmail.com>
References: 
 <CAJnjrPPXrdJOxeHuVHMhGmTLrpisLfZeoetj=F3o+qz8qZMeWA@mail.gmail.com>
 <CAL8PwkbENpC5yOqSqqeaS6JMQNBYWtcvdHXn7PKv=ScZdwDLAQ@mail.gmail.com>
From: Matthew Bellew <matthewb@labkey.com>
Date: Fri, 2 Mar 2012 15:56:49 -0800
Message-ID: 
 <CAJnjrPP+OSEx7=5=qTLgE21JYC6Q0EROtBEVDV3LEXJW67pjAQ@mail.gmail.com>
Subject: Re: CloseableThreadLocal problem
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks for your reply MIke, I create this bug.

https://issues.apache.org/jira/browse/LUCENE-3841

Matt

On Thu, Mar 1, 2012 at 2:32 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Phew, tricky.
>
> The problem is purging is potentially costly... it iterates all
> entries in the map (threads that have called get) looking for dead
> threads.
>
> Can you open an issue...? =C2=A0We can iterate there. =C2=A0Thanks for ra=
ising this,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Feb 29, 2012 at 12:17 PM, Matthew Bellew <matthewb@labkey.com> wr=
ote:
>> We tracked down a large memory leak (effectively a leak anyway) caused
>> by how Analyzer users CloseableThreadLocal.
>> CloseableThreadLocal.hardRefs holds references to Thread objects as
>> keys. =C2=A0The problem is that it only frees these references in the se=
t()
>> method, and SnowballAnalyzer will only call set() when it is used by a
>> NEW thread.
>>
>> The problem scenario is as follows:
>>
>> The server experiences a spike in usage (say by robots or whatever)
>> and many threads are created and referenced by
>> CloseableThreadLocal.hardRefs. =C2=A0The server quiesces and lets many o=
f
>> these threads expire normally. =C2=A0Now we have a smaller, but adequate
>> thread pool. =C2=A0So CloseableThreadLocal.set() may not be called by
>> SnowBallAnalyzer (via Analyzer) for a _long_ time. =C2=A0The purge code =
is
>> never called, and these threads along with their thread local storage
>> (lucene related or not) is never cleaned up.
>>
>> I think calling the purge code in both get() and set() would have
>> avoided this problem, perhaps using WeakHashMap instead of HashMap may
>> also have helped (WeakHashMap purges on get() and set())
>>
>> Our current work around is to not share SnowBallAnalyzer instances
>> among HTTP searcher threads. =C2=A0We open and close one on every reques=
t.
>>
>> Thanks,
>> Matt
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org