lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kristofer Karlsson (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4740) Weak references cause extreme GC churn
Date Fri, 01 Feb 2013 10:17:15 GMT


Kristofer Karlsson commented on LUCENE-4740:

bq. I agree that might be aproblem and you may be facing it. How mayn requests per second
do you have on your server?

Not that many - about 8000 per minute on yesterdays peak, which is about 133 per second. However,
each requests leads to several complex lucene queries, though I don't have any numbers on
the actual lucene query throughput.

bq. This behaviour is Java's weak reference GC behaviour, it has nothing to do with WeakIdentityMap.
The default WeakHashMap from JDK has the same problems.


bq. My ide was that the master creates some boolean[1] and passes this boolen[1] array to
all childs. When the master closes, it does set the b[0] to false. All childs would do a check
on b[0]... Not sure how this affects performance.

Yes, I thought about this too, and I am not sure the performance penalty would be that problematic
(but it would need to be measured). And if possibly, users of the inputs should avoid doing
small individual byte gets, and instead try to consume chunks of bytes to avoid the overhead.

I would prefer an AtomicBoolean since it uses a volatile field. As far as I know, you can't
make contents of arrays volatile.
In any case, wouldn't it would possible to skip the whole master/slave relationship and make
everyone equal, just sharing the closed state flag? Though then running close() on a clone
would close everything, which is possibly not what you want to happen.

> Weak references cause extreme GC churn
> --------------------------------------
>                 Key: LUCENE-4740
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/store
>    Affects Versions: 3.6.1
>         Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16 cores
>            Reporter: Kristofer Karlsson
>            Priority: Critical
>         Attachments: LUCENE-4740.patch
> We are running a set of independent search machines, running our custom software using
lucene as a search library. We recently upgraded from lucene 3.0.3 to 3.6.1 and noticed a
severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling because it's
spending so much time in GC. We noticed about 212 million WeakReference, originating from
WeakIdentityMap, originating from MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from MMapIndexInput,
and as a side-effect, disabling support for explictly unmapping the mmapped data.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message