Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Sun, 11 Mar 2012 14:01:58 +0000 (UTC)
From: "Michael McCandless (Commented) (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <1490248779.49080.1331474518776.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <758869398.15874.1330732677586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (LUCENE-3841) CloseableThreadLocal does not work
 well with Tomcat thread pooling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227049#comment-13227049 ] 

Michael McCandless commented on LUCENE-3841:
--------------------------------------------

I think it should be safe to use a WeakHashMap for the hardRefs instead of HashMap?

This way, if a thread has finished and its Thread object is otherwise GCable, the entries in hardRefs should be cleared... though, it's not clear to me precisely when they will be cleared.  If it's only on future access to the WeakHashMap (get or set), which seems likely because I think WeakHashMap uses a WeakReference for the keys and therefore won't really remove an entry util it's later "touched", then again only on set will the object be cleared and we haven't really improved the situation.

Matthew, did you try that change, and, did it improve the scenario above?

Failing that, I think we have to purge it get... maybe we can amortize it (every Nth get, where N is a factor of how many entries are in the map...).

Also: I don't think PagedBytes should use CloseableThreadLocal... I think it should just new byte[].

Separately: maybe SnowballAnalyzer is too heavy...?  Does it have some static data that ought to be loaded once and shared across analyzers... but isn't today?
                
> CloseableThreadLocal does not work well with Tomcat thread pooling
> ------------------------------------------------------------------
>
>                 Key: LUCENE-3841
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3841
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: core/other
>    Affects Versions: 3.5
>         Environment: Lucene/Tika/Snowball running in a Tomcat web application
>            Reporter: Matthew Bellew
>            Assignee: Michael McCandless
>             Fix For: 3.6, 4.0
>
>
> We tracked down a large memory leak (effectively a leak anyway) caused
> by how Analyzer users CloseableThreadLocal.
> CloseableThreadLocal.hardRefs holds references to Thread objects as
> keys.  The problem is that it only frees these references in the set()
> method, and SnowballAnalyzer will only call set() when it is used by a
> NEW thread.
> The problem scenario is as follows:
> The server experiences a spike in usage (say by robots or whatever)
> and many threads are created and referenced by
> CloseableThreadLocal.hardRefs.  The server quiesces and lets many of
> these threads expire normally.  Now we have a smaller, but adequate
> thread pool.  So CloseableThreadLocal.set() may not be called by
> SnowBallAnalyzer (via Analyzer) for a _long_ time.  The purge code is
> never called, and these threads along with their thread local storage
> (lucene related or not) is never cleaned up.
> I think calling the purge code in both get() and set() would have
> avoided this problem, but is potentially expensive.  Perhaps using 
> WeakHashMap instead of HashMap may also have helped.  WeakHashMap 
> purges on get() and set().  So this might be an efficient way to
> clean up threads in get(), while set() might do the more expensive
> Map.keySet() iteration.
> Our current work around is to not share SnowBallAnalyzer instances
> among HTTP searcher threads.  We open and close one on every request.
> Thanks,
> Matt

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org