Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <6579271.1146496486704.JavaMail.jira@brutus>
Date: Mon, 1 May 2006 15:14:46 +0000 (GMT+00:00)
From: "Nicholaus Shupe (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-529) TermInfosReader and other + instance
 ThreadLocal => transient/odd memory leaks =>  OutOfMemoryException
In-Reply-To: <131213190.1143047221584.JavaMail.jira@ajax>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/LUCENE-529?page=comments#action_12377225 ] 

Nicholaus Shupe commented on LUCENE-529:
----------------------------------------

I'm experiencing a memory leak in Lucene 1.9.1 that might be related to this problem.  Looks like I'll be creating my own patched version of Lucene in the meantime.

> TermInfosReader and other + instance ThreadLocal => transient/odd memory leaks =>  OutOfMemoryException
> -------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-529
>          URL: http://issues.apache.org/jira/browse/LUCENE-529
>      Project: Lucene - Java
>         Type: Bug

>   Components: Index
>     Versions: 1.9
>  Environment: Lucene 1.4.3 with 1.5.0_04 JVM or newer......will aplpy to 1.9 code 
>     Reporter: Andy Hind
>  Attachments: ThreadLocalTest.java
>
> TermInfosReader uses an instance level ThreadLocal for enumerators.
> This is a transient/odd memory leak in lucene 1.4.3-1.9 and applies to current JVMs, 
> not just an old JVM issue as described in the finalizer of the 1.9 code.
> There is also an instance level thread local in SegmentReader....which will have the same issue.
> There may be other uses which also need to be fixed.
> I don't understand the intended use for these variables.....however
> Each ThreadLocal has its own hashcode used for look up, see the ThreadLocal source code. Each instance of TermInfosReader will be creating an instance of the thread local. All this does is create an instance variable on each thread when it accesses the thread local. Setting it to null in the finaliser will set it to null on one thread, the finalizer thread, where it has never been created.  There is no point to this :-(
> I assume there is a good concurrency reason why an instance variable can not be used...
> I have not used multi-threaded searching, but I have used a lot of threads each making searchers and searching.
> 1.4.3 has a clear memory leak caused by this thread local. This use case above is definitely solved by setting the thread local to null in the close(). This at least has a chance of being on the correct thread :-) 
> I know reusing Searchers would help but that is my choice and I will get to that later .... 
> Now you wnat to know why....
> Thread locals are stored in a table of entries. Each entry is *weak reference* to the key (Here the TermInfosReader instance)  and a *simple reference* to the thread local value. When the instance is GCed its key becomes null. 
> This is now a stale entry in the table.
> Stale entries are cleared up in an ad hoc way and until they are cleared up the value will not be garbage collected.
> Until the instance is GCed it is a valid key and its presence may cause the table to expand.
> See the ThreadLocal code.
> So if you have lots of threads, all creating thread locals rapidly, you can get each thread holding a large table of thread locals which all contain many stale entries and preventing some objects from being garbage collected. 
> The limited GC of the thread local table is not enough to save you from running out of memory.  
> Summary:
> ========
> - remove finalizer()
> - set the thread local to null in close() 
>   - values will be available for gc 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org