accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Giordano <MGiord...@netcentricinc.com>
Subject RE: PermGen leak in JBoss
Date Fri, 03 May 2013 15:43:02 GMT
Sorry for resurrecting this 'old' thread but I need to open a ticket for this.

How do I do that ?

Thanks,
Mike G.

From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, April 24, 2013 12:23 PM
To: user@accumulo.apache.org
Subject: Re: PermGen leak in JBoss



On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <HAbelson@netcentricinc.com<mailto:HAbelson@netcentricinc.com>>
wrote:
Hi, I am a colleague of Michael's, I can provide more info

The following are the 5 threads that are not being cleaned up on an undeploy. It's hard to
get more stack info than this.

Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Running)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)

Using Plumbr, the report states that the SendThread and EventThread above both contain an
instance of org.apache.zookeeper.ClientCnxn


I tried this on 1.4 and then ran jstack using following program.


public class Test {
  public static void main(String[] args) throws Exception {
    ZooKeeperInstance instance = new ZooKeeperInstance("test14", "localhost");
    Connector connector = instance.getConnector("root", "secret");

    UtilWaitThread.sleep(600000);

  }
}

I see the following threads after filtering out java threads and the thread calling main.
 I am running this code in a different way.  I used the accumulo script to execute it, which
uses Accumulo start, which introduces the AccumuloFilesystemAlterationMonitor thread.  I am
seeing three of the threads you see, I am not seeing the FileWatchDog threads.

"Thrift Connection Pool Checker" daemon prio=10 tid=0x00007f861839f800 nid=0x61f0 waiting
on condition [0x00007f8652a62000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
            at java.lang.Thread.run(Thread.java:662)

"Test-EventThread" daemon prio=10 tid=0x00007f8618119000 nid=0x61ee waiting on condition [0x00007f8652d02000]
   java.lang.Thread.State: WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0x00000000bb402270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
            at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
            at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

"Test-SendThread(localhost:2181)" daemon prio=10 tid=0x00007f8618103000 nid=0x61ed runnable
[0x00007f8652e03000]
   java.lang.Thread.State: RUNNABLE
            at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
            at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
            - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)
            - locked <0x00000000bb3e0e78> (a java.util.Collections$UnmodifiableSet)
            - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl)
            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)

"org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor" daemon prio=10
tid=0x00007f86a4387000 nid=0x61e9 waiting on condition [0x00007f865367c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127)
            at java.lang.Thread.run(Thread.java:662)


Should probably open a ticket about this issue.  We could probably do something about the
Thrift Connection Pool Checker threads.   I am not sure if the zookeeper threads are an issue
w/ zookeeper or an issue w/ how accumulo is using zookeeper.


-Heath

-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org<mailto:user@accumulo.apache.org>
Subject: PermGen leak in JBoss

Under version 1.3.7 we are using the following code to initialize a cloudbase connection during
initialization of our web app:

                        ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
                        connector = instance.getConnector(userId, password.getBytes());

The problem is that under the hood, this call creates several (5) threads that are not cleaned
up when the app is undeployed in JBoss. This is occurring without performing any scans or
interacting with cloudbase in any other way. After relatively few redeploys of the app, the
PermGen Space is OOM.

I can't find any reference in the cloudbase API akin to a close() method for the Connector
object. This is a classloader leak effecting any webapp that is accessing cloudbase directly.
The result of this leak is not simply orphaned threads, but thousands of classes not gc'd
because the classloader itself can't be gc'd. This is what is filling up PermGen.

Has anyone discovered this particular issue? Has anyone discovered a solution?

Thanks,
Mike G.

NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ  07719
Phone: 732-544-0888<tel:732-544-0888>



Mime
View raw message