accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Heath Abelson <HAbel...@netcentricinc.com>
Subject RE: PermGen leak in JBoss
Date Wed, 24 Apr 2013 16:58:14 GMT
Nice, I'll try that. That's 1 out of 3 (I'm ignoring the Watchdogs for now)

I am able to kill the SendThread by blocking port 2181. I know that's not really a viable
workaround, I just thought it might be relevant info for tracking down the problem.

From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, April 24, 2013 12:35 PM
To: user@accumulo.apache.org
Subject: Re: PermGen leak in JBoss

If you execute the following code in the before the app is undeployed and in the correct classloader
context, then it will cause the "Thrift Connection Pool Checker" thread to die w/ a NPE being
printed.

    Field cacheField = ThriftTransportPool.class.getDeclaredField("cache");
    cacheField.setAccessible(true);
    cacheField.set(ThriftTransportPool.getInstance(), null);

On Wed, Apr 24, 2013 at 12:08 PM, Heath Abelson <HAbelson@netcentricinc.com<mailto:HAbelson@netcentricinc.com>>
wrote:
As an additional point, we have a second webapp that also uses 1.3.7 that also leaves the
same threads - minus the 2 two watchdog threads

From: Eric Newton [mailto:eric.newton@gmail.com<mailto:eric.newton@gmail.com>]
Sent: Wednesday, April 24, 2013 11:47 AM
To: user@accumulo.apache.org<mailto:user@accumulo.apache.org>
Subject: Re: PermGen leak in JBoss

One is for the dynamic reloading of the log4j configuration, IIRC.

-Eric


On Wed, Apr 24, 2013 at 11:44 AM, Keith Turner <keith@deenlo.com<mailto:keith@deenlo.com>>
wrote:


On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <HAbelson@netcentricinc.com<mailto:HAbelson@netcentricinc.com>>
wrote:
Hi, I am a colleague of Michael's, I can provide more info

The following are the 5 threads that are not being cleaned up on an undeploy. It's hard to
get more stack info than this.

Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Running)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)

Using Plumbr, the report states that the SendThread and EventThread above both contain an
instance of org.apache.zookeeper.ClientCnxn

So two zookeeper threads. An Accumulo thread (Thrift Connection Pool Checker).  What are the
FileWatchdog threads?  Those could be related to accumulo-start or hdfs.


-Heath

-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org<mailto:user@accumulo.apache.org>
Subject: PermGen leak in JBoss

Under version 1.3.7 we are using the following code to initialize a cloudbase connection during
initialization of our web app:

                        ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
                        connector = instance.getConnector(userId, password.getBytes());

The problem is that under the hood, this call creates several (5) threads that are not cleaned
up when the app is undeployed in JBoss. This is occurring without performing any scans or
interacting with cloudbase in any other way. After relatively few redeploys of the app, the
PermGen Space is OOM.

I can't find any reference in the cloudbase API akin to a close() method for the Connector
object. This is a classloader leak effecting any webapp that is accessing cloudbase directly.
The result of this leak is not simply orphaned threads, but thousands of classes not gc'd
because the classloader itself can't be gc'd. This is what is filling up PermGen.

Has anyone discovered this particular issue? Has anyone discovered a solution?

Thanks,
Mike G.

NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ  07719
Phone: 732-544-0888<tel:732-544-0888>




Mime
View raw message