lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kristofer Karlsson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4740) Weak references cause extreme GC churn
Date Thu, 31 Jan 2013 20:33:12 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568074#comment-13568074
] 

Kristofer Karlsson commented on LUCENE-4740:
--------------------------------------------

After doing some more thinking and micro-benchmarking, I think the problem occurs when you
create clones at a faster rate than the GC can cope with.
{noformat}
public class WeakTest extends TestCase {
    private static final int CPUS = Runtime.getRuntime().availableProcessors();

    public void testFoo() throws Exception {
        final WeakIdentityMap<Object, Boolean> map = WeakIdentityMap.newConcurrentHashMap();
        ExecutorService executorService = Executors.newCachedThreadPool();

        for (int k = 0; k < CPUS; k++) {
            executorService.submit(new Runnable() {
                public void run() {
                    while (true) {
                        map.put(new Object(), Boolean.TRUE);
                    }
                }
            });
            executorService.submit(new Runnable() {
                public void run() {
                    while (true) {
                        System.gc();
                    }
                }
            });
        }
        while (true) {
            System.out.println("Map size: " + map.size());
            Thread.sleep(1000);
        }
    }

}
{noformat}
I tried running this with -Xmx100m. This makes the map grow indefinitely.
I know this is not a very reliable test, since the JVM only considers System.gc() a hint,
but it definitely seems like it's not very good at freeing weak references during high load.

If I add Thread.sleep(1) in the put-worker, I can make it grow slower, but it still seems
to grow over time.

(Running java 7 on my home computer)
{noformat}
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
{noformat}

                
> Weak references cause extreme GC churn
> --------------------------------------
>
>                 Key: LUCENE-4740
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4740
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/store
>    Affects Versions: 3.6.1
>         Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16 cores
>            Reporter: Kristofer Karlsson
>            Priority: Critical
>
> We are running a set of independent search machines, running our custom software using
lucene as a search library. We recently upgraded from lucene 3.0.3 to 3.6.1 and noticed a
severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling because it's
spending so much time in GC. We noticed about 212 million WeakReference, originating from
WeakIdentityMap, originating from MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from MMapIndexInput,
and as a side-effect, disabling support for explictly unmapping the mmapped data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message