lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: WeakIdentityMap high memory usage
Date Wed, 07 Aug 2013 13:45:28 GMT
This map is used to track all cloned open files, which can be a very
large number over time (each search will create maybe 3 of them).

This is done as a "best effort" to prevent SEGV (JVM dies) if you
accidentally try to use an IndexReader after it was closed, while
using MMapDirectory.

However, it's a weak map, which means when HEAP is tight GC should drop it.

So, this should not cause a real problem in "real life", even though
it looks scary when you look at its RAM usage under a profiler.

If somehow it's causing "real life" problems, please report back!  But
a simple workaround is to call MMapDirectory.setUseUnmap(false) to
turn off this tracking; this means you rely on GC to (eventually)
unmap.

Mike McCandless

http://blog.mikemccandless.com


On Wed, Aug 7, 2013 at 2:45 AM, Denis Bazhenov <bazhenov@farpost.com> wrote:
> We have upgraded from Lucene 3.6 to 4.4.On the production we faced high minor GC time.
Heap dump showed that one of the biggest objects by size is org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference.
About 11 million instances with about 377 megabytes of memory in total (this is not even retained
size). Here is screenshot of the JProfiler output: https://dl.dropboxusercontent.com/u/16254496/Screen%20Shot%202013-08-07%20at%205.35.22%20PM.png.
>
> The keys of the map are MMapIndexInput. What this map is for and how can I reduce it
memory usage?
> ---
> Denis Bazhenov <bazhenov@farpost.com>
> FarPost.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message