lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alon Muchnick <a...@datonics.com>
Subject Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .
Date Thu, 03 Jan 2013 17:16:08 GMT
hi Mike ,

at the peak there are 500 live threads  going through Lucune (not all of
them at the same time , tomcat thread pool  uses round robin ) ,regarding
the Directory impl we are using RAMDirectory.
the object that takes most of the heap  is the "hardRefs" WeakHashMap class
member in the CloseableThreadLocal class . the size of the map is 500 ,
with one entry for each thread that went through Lucune.

when in run the test with only one thread the initial RAM did not grow much
beyond the initial 30MB .

when looking at the code i can see that the references to the hardRefs map
will be cleared when :

1.a thread which searched Lucune is no longer alive , so its corresponding
record in the map will be cleared either by the GC or by the purge() method
.

2.the close() method is called which "should only be called when all
threads are done using the instance" .

what happens to threads that go through Lucune and stay alive for a very
long times ?
will they always be a reference key in the hardRefs map for them ?

if so does the value for the corresponding record in the map will be
overwritten each time this thread makes a new search ? or will it be some
how accumulated ? :

value ---- org.apache.lucene.index.TermInfosReader$ThreadResources --->

termInfoCache  |org.apache.lucene.util.cache.SimpleLRUCache
termEnum       |org.apache.lucene.index.SegmentTermEnum

i have reduced the size of the thread pool , and changed to a more
aggressive thread closing policy ,threads are now reduced when they are not
needed (before that the thread pool would reach its max size and stayed
there),
this should kick in the the purge method and reclaim memory from the closed
threads .

ill give the system a few hours to run and update on the results .

Alon

















On Thu, Jan 3, 2013 at 2:58 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> then see if RAM doesn't grow much beyond that initial 30 MB?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message