Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 00E1FF0BE for ; Fri, 3 May 2013 15:42:19 +0000 (UTC) Received: (qmail 70718 invoked by uid 500); 3 May 2013 15:42:18 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 70673 invoked by uid 500); 3 May 2013 15:42:18 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 70665 invoked by uid 99); 3 May 2013 15:42:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 May 2013 15:42:18 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: encountered temporary error during SPF processing of domain of MGiordano@netcentricinc.com) Received: from [74.93.212.105] (HELO shark6.netcentric.local) (74.93.212.105) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 May 2013 15:42:14 +0000 Received: from SHARK6.netcentric.local ([::1]) by shark6.netcentric.local ([::1]) with mapi id 14.01.0218.012; Fri, 3 May 2013 11:43:04 -0400 From: Michael Giordano To: "user@accumulo.apache.org" Subject: RE: PermGen leak in JBoss Thread-Topic: PermGen leak in JBoss Thread-Index: Ac5A8ld6eCRds1RDQmChsPV5tMGkMAADNfuQAAqU74ABusn4cA== Date: Fri, 3 May 2013 15:43:02 +0000 Message-ID: <1F768E0365ED884CA8D9A3A315D04401084C4057@shark6.netcentric.local> References: <1F768E0365ED884CA8D9A3A315D04401084B1A92@shark6.netcentric.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.2.139] Content-Type: multipart/alternative; boundary="_000_1F768E0365ED884CA8D9A3A315D04401084C4057shark6netcentri_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_1F768E0365ED884CA8D9A3A315D04401084C4057shark6netcentri_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Sorry for resurrecting this 'old' thread but I need to open a ticket for th= is. How do I do that ? Thanks, Mike G. From: Keith Turner [mailto:keith@deenlo.com] Sent: Wednesday, April 24, 2013 12:23 PM To: user@accumulo.apache.org Subject: Re: PermGen leak in JBoss On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson > wrote: Hi, I am a colleague of Michael's, I can provide more info The following are the 5 threads that are not being cleaned up on an undeplo= y. It's hard to get more stack info than this. Daemon Thread [FileWatchdog] (Running) Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Runnin= g) Daemon Thread [MSC service thread 1-5-EventThread] (Running) Daemon Thread [Thrift Connection Pool Checker] (Running) Daemon Thread [FileWatchdog] (Running) Using Plumbr, the report states that the SendThread and EventThread above b= oth contain an instance of org.apache.zookeeper.ClientCnxn I tried this on 1.4 and then ran jstack using following program. public class Test { public static void main(String[] args) throws Exception { ZooKeeperInstance instance =3D new ZooKeeperInstance("test14", "localho= st"); Connector connector =3D instance.getConnector("root", "secret"); UtilWaitThread.sleep(600000); } } I see the following threads after filtering out java threads and the thread= calling main. I am running this code in a different way. I used the accu= mulo script to execute it, which uses Accumulo start, which introduces the = AccumuloFilesystemAlterationMonitor thread. I am seeing three of the threa= ds you see, I am not seeing the FileWatchDog threads. "Thrift Connection Pool Checker" daemon prio=3D10 tid=3D0x00007f861839f800 = nid=3D0x61f0 waiting on condition [0x00007f8652a62000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.accumulo.core.client.impl.ThriftTransportPool$Clo= ser.run(ThriftTransportPool.java:129) at java.lang.Thread.run(Thread.java:662) "Test-EventThread" daemon prio=3D10 tid=3D0x00007f8618119000 nid=3D0x61ee w= aiting on condition [0x00007f8652d02000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000bb402270> (a java.util.concur= rent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java= :156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Condit= ionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlocking= Queue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.j= ava:498) "Test-SendThread(localhost:2181)" daemon prio=3D10 tid=3D0x00007f8618103000= nid=3D0x61ed runnable [0x00007f8652e03000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210= ) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java= :65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69= ) - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2) - locked <0x00000000bb3e0e78> (a java.util.Collections$Unmodifi= ableSet) - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.ja= va:1131) "org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor"= daemon prio=3D10 tid=3D0x00007f86a4387000 nid=3D0x61e9 waiting on conditio= n [0x00007f865367c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.accumulo.start.classloader.AccumuloFilesystemAlte= rationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127) at java.lang.Thread.run(Thread.java:662) Should probably open a ticket about this issue. We could probably do somet= hing about the Thrift Connection Pool Checker threads. I am not sure if t= he zookeeper threads are an issue w/ zookeeper or an issue w/ how accumulo = is using zookeeper. -Heath -----Original Message----- From: Michael Giordano Sent: Wednesday, April 24, 2013 9:48 AM To: user@accumulo.apache.org Subject: PermGen leak in JBoss Under version 1.3.7 we are using the following code to initialize a cloudba= se connection during initialization of our web app: ZooKeeperInstance instance =3D new ZooKeeperInstanc= e(instanceName, zooKeepers); connector =3D instance.getConnector(userId, passwor= d.getBytes()); The problem is that under the hood, this call creates several (5) threads t= hat are not cleaned up when the app is undeployed in JBoss. This is occurri= ng without performing any scans or interacting with cloudbase in any other = way. After relatively few redeploys of the app, the PermGen Space is OOM. I can't find any reference in the cloudbase API akin to a close() method fo= r the Connector object. This is a classloader leak effecting any webapp tha= t is accessing cloudbase directly. The result of this leak is not simply or= phaned threads, but thousands of classes not gc'd because the classloader i= tself can't be gc'd. This is what is filling up PermGen. Has anyone discovered this particular issue? Has anyone discovered a soluti= on? Thanks, Mike G. NetCentric Technology, Inc. 3349 Route 138, Building A Wall, NJ 07719 Phone: 732-544-0888 --_000_1F768E0365ED884CA8D9A3A315D04401084C4057shark6netcentri_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Sorry for resurrecting th= is ‘old’ thread but I need to open a ticket for this.

 <= /p>

How do I do that ?

 <= /p>

Thanks,=

Mike G.=

 <= /p>

From: Keith Tu= rner [mailto:keith@deenlo.com]
Sent: Wednesday, April 24, 2013 12:23 PM
To: user@accumulo.apache.org
Subject: Re: PermGen leak in JBoss

 

 

 

On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <= HAbelson@ne= tcentricinc.com> wrote:

Hi, I am a colleague of Michael's, I can provide mor= e info

The following are the 5 threads that are not being cleaned up on an undeplo= y. It's hard to get more stack info than this.

Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Runnin= g)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)

Using Plumbr, the report states that the SendThread and EventThread above b= oth contain an instance of org.apache.zookeeper.ClientCnxn

 

 

I tried this on 1.4 and then ran jstack using follow= ing program.  

 

 

public class Test {

  public static void main(String[] args) throws= Exception {

    ZooKeeperInstance instance =3D new Zoo= KeeperInstance("test14", "localhost");

    Connector connector =3D instance.getCo= nnector("root", "secret");

    

    UtilWaitThread.sleep(600000);

 

  }

}

 

I see the following threads after filtering out java= threads and the thread calling main.  I am running this code in a dif= ferent way.  I used the accumulo script to execute it, which uses Accu= mulo start, which introduces the AccumuloFilesystemAlterationMonitor thread.  I am seeing three of the threads you see, I am not seeing th= e FileWatchDog threads.

 

"Thrift Connection Pool Checker" daemon pr= io=3D10 tid=3D0x00007f861839f800 nid=3D0x61f0 waiting on condition [0x00007= f8652a62000]

   java.lang.Thread.State: TIMED_WAITING (= sleeping)

        &nbs= p;   at java.lang.Thread.sleep(Native Method)

        &nbs= p;   at org.apache.accumulo.core.client.impl.ThriftTransportPool$= Closer.run(ThriftTransportPool.java:129)

        &nbs= p;   at java.lang.Thread.run(Thread.java:662)

 

"Test-EventThread" daemon prio=3D10 tid=3D= 0x00007f8618119000 nid=3D0x61ee waiting on condition [0x00007f8652d02000]

   java.lang.Thread.State: WAITING (parkin= g)

        &nbs= p;   at sun.misc.Unsafe.park(Native Method)

        &nbs= p;   - parking to wait for  <0x00000000bb402270> (a ja= va.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        &nbs= p;   at java.util.concurrent.locks.LockSupport.park(LockSupport.j= ava:156)

        &nbs= p;   at java.util.concurrent.locks.AbstractQueuedSynchronizer$Con= ditionObject.await(AbstractQueuedSynchronizer.java:1987)

        &nbs= p;   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlock= ingQueue.java:399)

        &nbs= p;   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnx= n.java:498)

 

"Test-SendThread(localhost:2181)" daemon p= rio=3D10 tid=3D0x00007f8618103000 nid=3D0x61ed runnable [0x00007f8652e03000= ]

   java.lang.Thread.State: RUNNABLE

        &nbs= p;   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        &nbs= p;   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:= 210)

        &nbs= p;   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.j= ava:65)

        &nbs= p;   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java= :69)

        &nbs= p;   - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)

        &nbs= p;   - locked <0x00000000bb3e0e78> (a java.util.Collections= $UnmodifiableSet)

        &nbs= p;   - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelec= torImpl)

        &nbs= p;   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)=

        &nbs= p;   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn= .java:1131)

 

"org.apache.accumulo.start.classloader.Accumulo= FilesystemAlterationMonitor" daemon prio=3D10 tid=3D0x00007f86a4387000= nid=3D0x61e9 waiting on condition [0x00007f865367c000]

   java.lang.Thread.State: TIMED_WAITING (= sleeping)

        &nbs= p;   at java.lang.Thread.sleep(Native Method)

        &nbs= p;   at org.apache.accumulo.start.classloader.AccumuloFilesystemA= lterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127)

        &nbs= p;   at java.lang.Thread.run(Thread.java:662)

 

 

Should probably open a ticket about this issue. &nbs= p;We could probably do something about the Thrift Connection Pool Chec= ker threads.   I am not sure if the zookeeper threads are an issue w/ = zookeeper or an issue w/ how accumulo is using zookeeper.

 


-Heath


-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org
Subject: PermGen leak in JBoss

Under version 1.3.7 we are using the following code to initialize a cloudba= se connection during initialization of our web app:

                     = ;   ZooKeeperInstance instance =3D new ZooKeeperInstance(instanceName,= zooKeepers);
                     = ;   connector =3D instance.getConnector(userId, password.getBytes());<= br>
The problem is that under the hood, this call creates several (5) threads t= hat are not cleaned up when the app is undeployed in JBoss. This is occurri= ng without performing any scans or interacting with cloudbase in any other = way. After relatively few redeploys of the app, the PermGen Space is OOM.

I can't find any reference in the cloudbase API akin to a close() method fo= r the Connector object. This is a classloader leak effecting any webapp tha= t is accessing cloudbase directly. The result of this leak is not simply or= phaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is wha= t is filling up PermGen.

Has anyone discovered this particular issue? Has anyone discovered a soluti= on?

Thanks,
Mike G.

NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ  07719
Phone: 732-544-0888

 

--_000_1F768E0365ED884CA8D9A3A315D04401084C4057shark6netcentri_--