Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: error (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <A6EDF356-0D35-47F2-A3AC-406BA381AD63@farpost.com>
References: <A6EDF356-0D35-47F2-A3AC-406BA381AD63@farpost.com>
From: Michael McCandless <lucene@mikemccandless.com>
Date: Wed, 7 Aug 2013 09:45:28 -0400
Message-ID: 
 <CAL8PwkZ5tzkcWg9K6QidsvfsLrcZ9H0Y9Ls1mVUteBeYpiNpXA@mail.gmail.com>
Subject: Re: WeakIdentityMap high memory usage
To: Lucene Users <java-user@lucene.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

This map is used to track all cloned open files, which can be a very
large number over time (each search will create maybe 3 of them).

This is done as a "best effort" to prevent SEGV (JVM dies) if you
accidentally try to use an IndexReader after it was closed, while
using MMapDirectory.

However, it's a weak map, which means when HEAP is tight GC should drop it.

So, this should not cause a real problem in "real life", even though
it looks scary when you look at its RAM usage under a profiler.

If somehow it's causing "real life" problems, please report back!  But
a simple workaround is to call MMapDirectory.setUseUnmap(false) to
turn off this tracking; this means you rely on GC to (eventually)
unmap.

Mike McCandless

http://blog.mikemccandless.com


On Wed, Aug 7, 2013 at 2:45 AM, Denis Bazhenov <bazhenov@farpost.com> wrote=
:
> We have upgraded from Lucene 3.6 to 4.4.On the production we faced high m=
inor GC time. Heap dump showed that one of the biggest objects by size is o=
rg.apache.lucene.util.WeakIdentityMap$IdentityWeakReference. About 11 milli=
on instances with about 377 megabytes of memory in total (this is not even =
retained size). Here is screenshot of the JProfiler output: https://dl.drop=
boxusercontent.com/u/16254496/Screen%20Shot%202013-08-07%20at%205.35.22%20P=
M.png.
>
> The keys of the map are MMapIndexInput. What this map is for and how can =
I reduce it memory usage?
> ---
> Denis Bazhenov <bazhenov@farpost.com>
> FarPost.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org