curator-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig McNally (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CURATOR-167) Memory leak in NodeCache
Date Tue, 18 Nov 2014 16:12:34 GMT
Craig McNally created CURATOR-167:
-------------------------------------

             Summary: Memory leak in NodeCache
                 Key: CURATOR-167
                 URL: https://issues.apache.org/jira/browse/CURATOR-167
             Project: Apache Curator
          Issue Type: Bug
          Components: Client
    Affects Versions: 2.6.0
         Environment: Linux 3.13.0-37-generic #64~precise1-Ubuntu SMP Wed Sep 24 21:39:43
UTC 2014 i686 i686 i386 GNU/Linux

java version "1.7.0_13"
Java(TM) SE Runtime Environment (build 1.7.0_13-b20)
Java HotSpot(TM) Server VM (build 23.7-b01, mixed mode)

            Reporter: Craig McNally
            Priority: Critical


There's a memory leak in NodeCache.  I was able to reliably reproduce the problem using a
very simple test that performs the following:

1) Creates a CuratorFramework instance and starts it.
2) in a loop:  Creates a NodeCache and starts it, then closes it.

Eventually you get a java.lang.OutOfMemoryError:  Java heap space.  This happens regardless
of the heap size, though it happens much faster with a small heap.

Upon furher investigation w/ a profiler, I can see that each NodeCache is being referenced
by the NamespaceWatcherMap.

Here's the test code:

{code:title=NodeCacheLeakTest.java|borderStyle=solid}
import java.util.Date;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.cache.NodeCache;
import org.apache.curator.retry.ExponentialBackoffRetry;

public class NodeCacheLeakTest {

	public static void main(String[] args) throws Exception {

		String zkConnect;
		if (args.length > 0 && args[0] != null)
			zkConnect = args[0];
		else
			zkConnect = "localhost:2181/test";

		CuratorFramework curator = CuratorFrameworkFactory.newClient(zkConnect,
				new ExponentialBackoffRetry(500, 10));
		curator.start();

		int count = 0;
		while (true) {
			String nodePath = "/foo/node-" + (count);

			NodeCache cache = new NodeCache(curator, nodePath);
			cache.start(true);
			cache.close();

			count++;

			if (count % 1000 == 0)
				System.out.println(new Date() + " Started and Closed " + count
						+ " NodeCache instances");
		}
	}
}
{code}

And here's the output/OOM Error when using (-Xms10m -Xmx10m -XX:+UseG1GC):
{noformat}
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Tue Nov 18 15:59:12 GMT+00:00 2014 Started and Closed 1000 NodeCache instances
Tue Nov 18 15:59:16 GMT+00:00 2014 Started and Closed 2000 NodeCache instances
Tue Nov 18 15:59:20 GMT+00:00 2014 Started and Closed 3000 NodeCache instances
Tue Nov 18 15:59:23 GMT+00:00 2014 Started and Closed 4000 NodeCache instances
Tue Nov 18 15:59:27 GMT+00:00 2014 Started and Closed 5000 NodeCache instances
Tue Nov 18 15:59:31 GMT+00:00 2014 Started and Closed 6000 NodeCache instances
Tue Nov 18 15:59:36 GMT+00:00 2014 Started and Closed 7000 NodeCache instances
Tue Nov 18 15:59:40 GMT+00:00 2014 Started and Closed 8000 NodeCache instances
Tue Nov 18 15:59:45 GMT+00:00 2014 Started and Closed 9000 NodeCache instances
Tue Nov 18 15:59:49 GMT+00:00 2014 Started and Closed 10000 NodeCache instances
Tue Nov 18 16:00:01 GMT+00:00 2014 Started and Closed 11000 NodeCache instances
java.lang.OutOfMemoryError: Java heap space
	at java.io.BufferedWriter.<init>(BufferedWriter.java:105)
	at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
	at java.io.PrintStream.<init>(PrintStream.java:112)
	at java.io.PrintStream.<init>(PrintStream.java:175)
	at org.apache.jute.CsvOutputArchive.<init>(CsvOutputArchive.java:57)
	at org.apache.zookeeper.proto.RequestHeader.toString(RequestHeader.java:62)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$Packet.toString(ClientCnxn.java:308)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:815)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
java.lang.OutOfMemoryError: Java heap space
	at java.io.BufferedWriter.<init>(BufferedWriter.java:105)
	at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
	at java.io.PrintStream.<init>(PrintStream.java:112)
	at java.io.PrintStream.<init>(PrintStream.java:175)
	at org.apache.jute.CsvOutputArchive.<init>(CsvOutputArchive.java:57)
	at org.apache.zookeeper.proto.RequestHeader.toString(RequestHeader.java:62)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$Packet.toString(ClientCnxn.java:308)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:815)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
java.lang.OutOfMemoryError: Java heap space
	at java.io.BufferedWriter.<init>(BufferedWriter.java:105)
	at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
	at java.io.PrintStream.<init>(PrintStream.java:112)
	at java.io.PrintStream.<init>(PrintStream.java:175)
	at org.apache.jute.CsvOutputArchive.<init>(CsvOutputArchive.java:57)
	at org.apache.zookeeper.proto.RequestHeader.toString(RequestHeader.java:62)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$Packet.toString(ClientCnxn.java:308)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:815)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
java.lang.OutOfMemoryError: Java heap space
	at java.io.BufferedWriter.<init>(BufferedWriter.java:105)
	at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
	at java.io.PrintStream.<init>(PrintStream.java:112)
	at java.io.PrintStream.<init>(PrintStream.java:175)
	at org.apache.jute.CsvOutputArchive.<init>(CsvOutputArchive.java:57)
	at org.apache.zookeeper.proto.RequestHeader.toString(RequestHeader.java:62)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$Packet.toString(ClientCnxn.java:308)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:815)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
{noformat}

Additional Notes:
* When I don't specify the G1 collector I still get OutOfMemoryErrors, only the cause is "GC
overhead limit exceeded".  {noformat}java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.io.BufferedWriter.<init>(BufferedWriter.java:105)
	at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
	at java.io.PrintStream.<init>(PrintStream.java:112)
	at java.io.PrintStream.<init>(PrintStream.java:175)
	at org.apache.jute.CsvOutputArchive.<init>(CsvOutputArchive.java:57)
	at org.apache.zookeeper.proto.GetDataRequest.toString(GetDataRequest.java:62)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$Packet.toString(ClientCnxn.java:310)
	at java.lang.String.valueOf(String.java:2854)
	at java.lang.StringBuilder.append(StringBuilder.java:128)
	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:815)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068){noformat}
* I have profiler screenshots but I'm not sure how to attach/upload those here...
* This has been around for a while as I first noticed it with v1.3.3.  I upgraded to v2.6.0
and re-ran the test, but as you can see this still appears to be an issue.
* I ran this test on a x86_64 Ubuntu 10.043 system and got the observed the same behavior




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message