lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4740) Weak references cause extreme GC churn
Date Fri, 01 Feb 2013 10:21:16 GMT


Uwe Schindler commented on LUCENE-4740:

bq. but what happens if you start with having useUnmap = false, then creating a bunch of clones,
and then setting it back to useUnmap = true? If I read the code correctly (which I am not
certain of), closing the original input will then unmap the data and break all the existing

The settings are decoupled:
If you start with useUnmap = false, all IndexInputs created will have no weak map, so when
they are closed, the clones are not touched.

If you change the setting to true, the already existing indexinputs will not be tracked (as
before), but new indexinputs will get a map and all of their clones will be freed correctly.

The other special case: If you change the setting from true -> false, all existing IndexInputs
will keep their maps and will be cleaned up on close (buffers set to null). But the cleanMapping()
method will get a no-op, so they are correctly nulled, but no longer unmapped.

In any case a SIGSEGV is prevented (as good as we can without locking).

In general, nothing breaks if you change the setting later, but you should really do it only
after construction.
> Weak references cause extreme GC churn
> --------------------------------------
>                 Key: LUCENE-4740
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/store
>    Affects Versions: 3.6.1
>         Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16 cores
>            Reporter: Kristofer Karlsson
>            Priority: Critical
>         Attachments: LUCENE-4740.patch
> We are running a set of independent search machines, running our custom software using
lucene as a search library. We recently upgraded from lucene 3.0.3 to 3.6.1 and noticed a
severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling because it's
spending so much time in GC. We noticed about 212 million WeakReference, originating from
WeakIdentityMap, originating from MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from MMapIndexInput,
and as a side-effect, disabling support for explictly unmapping the mmapped data.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message