lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: [jira] Updated: (LUCENE-529) TermInfosReader and other + instance ThreadLocal => transient/odd memory leaks => OutOfMemoryException
Date Thu, 23 Mar 2006 16:04:58 GMT
The only other thing that may be causing your problem is the use of finalize(). This can interfere
with the GC ability to GC objects.

I am not sure why the finalize() is used in the Lucene ThreadLocal handling. It doesn't seem
necessary to me.

-----Original Message-----
From: Robert Engels [mailto:rengels@ix.netcom.com]
Sent: Thursday, March 23, 2006 10:00 AM
To: java-dev@lucene.apache.org
Subject: RE: [jira] Updated: (LUCENE-529) TermInfosReader and other +
instance ThreadLocal => transient/odd memory leaks =>
OutOfMemoryException
Importance: High


Your testcase is invalid.

Reduce the size by 10, increase the repeat by 10, (SAME amount of memory use), and it works
fine.

The reason it works in the one case is that you use new WeakReference(new Arrary()), - since
the array cannot be referenced, it is immediately GC'd. You should have noticed since there
were no finalization messages printed. You can demonstrate this clearly by adding an else
to the if in the finalize() to print out that the object was indeed finalized.

ThreadLocal's work and are GC'd correctly.

There is something else wrong in your system.

I ran the test using 1.5.0._06.


-----Original Message-----
From: Andy Hind (JIRA) [mailto:jira@apache.org]
Sent: Thursday, March 23, 2006 7:07 AM
To: java-dev@lucene.apache.org
Subject: [jira] Updated: (LUCENE-529) TermInfosReader and other +
instance ThreadLocal => transient/odd memory leaks =>
OutOfMemoryException


     [ http://issues.apache.org/jira/browse/LUCENE-529?page=all ]

Andy Hind updated LUCENE-529:
-----------------------------

    Attachment: ThreadLocalTest.java

Attached is a test which you can use to see how ThreadLocals are left around.
Getting an out of memory exception depends on a number things....it is set up to fail for
64M

Now I understand what is going on, there are a few alternatives:

1) set null on close
- fine for single thread use
- probably leaves (n-1)*segments*2    things hanging around for n threaded use

2) Use a weak reference and leave it up to GC to get rid of the referent when it is not being
used

3) Manage the things youself by object id and thread id - and clean up on object close() 

I would go with option 1) and 2) although it may mean things get GCed before a call to close()
when not used.

The fix I initially suggested is in production, and has been stress tested with a couple of
hundred users continually pounding the app,
 but not for multithreaded use of IndexReaders. Each time does a couple of simple searches
with no clever reuse of index readers (which is on the todo list)

I do not see how setting the thread local to null on close() has any negative impact. You
are not going to use the cached information again??

Before the fix: 10-100 threads - 1G JVM - OOM in a few hours 
After: 10-100 threads 256M JVM -  days with a flat memory footprint

I am not sure why the thread local table is so big for us, but that is not really the issue.
It could just be building lots of IndexReaders (with thread locals hanging - probably making
10/instance ) and gc not kicking in so this table grows and can hold a lot of stale entries.
 I may get time to investigate further

> TermInfosReader and other + instance ThreadLocal => transient/odd memory leaks =>
 OutOfMemoryException
> -------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-529
>          URL: http://issues.apache.org/jira/browse/LUCENE-529
>      Project: Lucene - Java
>         Type: Bug
>   Components: Index
>     Versions: 1.9
>  Environment: Lucene 1.4.3 with 1.5.0_04 JVM or newer......will aplpy to 1.9 code 
>     Reporter: Andy Hind
>  Attachments: ThreadLocalTest.java
>
> TermInfosReader uses an instance level ThreadLocal for enumerators.
> This is a transient/odd memory leak in lucene 1.4.3-1.9 and applies to current JVMs,

> not just an old JVM issue as described in the finalizer of the 1.9 code.
> There is also an instance level thread local in SegmentReader....which will have the
same issue.
> There may be other uses which also need to be fixed.
> I don't understand the intended use for these variables.....however
> Each ThreadLocal has its own hashcode used for look up, see the ThreadLocal source code.
Each instance of TermInfosReader will be creating an instance of the thread local. All this
does is create an instance variable on each thread when it accesses the thread local. Setting
it to null in the finaliser will set it to null on one thread, the finalizer thread, where
it has never been created.  There is no point to this :-(
> I assume there is a good concurrency reason why an instance variable can not be used...
> I have not used multi-threaded searching, but I have used a lot of threads each making
searchers and searching.
> 1.4.3 has a clear memory leak caused by this thread local. This use case above is definitely
solved by setting the thread local to null in the close(). This at least has a chance of being
on the correct thread :-) 
> I know reusing Searchers would help but that is my choice and I will get to that later
.... 
> Now you wnat to know why....
> Thread locals are stored in a table of entries. Each entry is *weak reference* to the
key (Here the TermInfosReader instance)  and a *simple reference* to the thread local value.
When the instance is GCed its key becomes null. 
> This is now a stale entry in the table.
> Stale entries are cleared up in an ad hoc way and until they are cleared up the value
will not be garbage collected.
> Until the instance is GCed it is a valid key and its presence may cause the table to
expand.
> See the ThreadLocal code.
> So if you have lots of threads, all creating thread locals rapidly, you can get each
thread holding a large table of thread locals which all contain many stale entries and preventing
some objects from being garbage collected. 
> The limited GC of the thread local table is not enough to save you from running out of
memory.  
> Summary:
> ========
> - remove finalizer()
> - set the thread local to null in close() 
>   - values will be available for gc 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message