Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7CAA2F15E for ; Fri, 3 May 2013 15:54:53 +0000 (UTC) Received: (qmail 4955 invoked by uid 500); 3 May 2013 15:54:53 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 4874 invoked by uid 500); 3 May 2013 15:54:53 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 4865 invoked by uid 99); 3 May 2013 15:54:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 May 2013 15:54:53 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [209.85.220.182] (HELO mail-vc0-f182.google.com) (209.85.220.182) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 May 2013 15:54:48 +0000 Received: by mail-vc0-f182.google.com with SMTP id ia10so1562451vcb.27 for ; Fri, 03 May 2013 08:54:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=gm4mDKsaN56Jd1xqTLywDPNdmB27FKrNjFo0PUMFwzw=; b=dQ/W5TvwCJh9FGOBx28WUsmU1cQePEkg3+xmS+yJYYJ6h53+x8Rq0IUqSucvC9MzrR ukKMp4UJvz3JtxNiBwedW6zym9Kps0lCyMOJEjUZiyPzRo8BZA9eiUA9L2JHNOQe53i/ Gcvm77OP8R8Hwy13OKUzQRa9zhdvalI4Aj8M13FTTu56fcFEepFJPOtYeZdQC9XW0KDK e3klbSh+S5sev6MWUGb35wM7ftiK8FeylG0GSshgeF3CinZoUfcJ1hw3i//Kf6l9Bqfu CigJDqjveLDwTY7htAWQwG5+IiUMCdnz8izlr33d2a6X1dtyRbx8WQ+egOI6Rm+W7SiB jv2Q== MIME-Version: 1.0 X-Received: by 10.52.18.206 with SMTP id y14mr3304499vdd.29.1367596447044; Fri, 03 May 2013 08:54:07 -0700 (PDT) Received: by 10.220.236.207 with HTTP; Fri, 3 May 2013 08:54:06 -0700 (PDT) In-Reply-To: <1F768E0365ED884CA8D9A3A315D04401084C4057@shark6.netcentric.local> References: <1F768E0365ED884CA8D9A3A315D04401084B1A92@shark6.netcentric.local> <1F768E0365ED884CA8D9A3A315D04401084C4057@shark6.netcentric.local> Date: Fri, 3 May 2013 11:54:06 -0400 Message-ID: Subject: Re: PermGen leak in JBoss From: Keith Turner To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=bcaec50409ae6c261904dbd25cd7 X-Gm-Message-State: ALoCoQmqOlQ8fEjUX4GudguqyoecPLZSOwXKLE49xACQD55+3QsesEbQ/yPlri+JTDOt95SaosSF X-Virus-Checked: Checked by ClamAV on apache.org --bcaec50409ae6c261904dbd25cd7 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Fri, May 3, 2013 at 11:43 AM, Michael Giordano < MGiordano@netcentricinc.com> wrote: > Sorry for resurrecting this =91old=92 thread but I need to open a ticket= for > this. > https://issues.apache.org/jira/browse/ACCUMULO > **** > > ** ** > > How do I do that ?**** > > ** ** > > Thanks,**** > > Mike G.**** > > ** ** > > *From:* Keith Turner [mailto:keith@deenlo.com] > *Sent:* Wednesday, April 24, 2013 12:23 PM > > *To:* user@accumulo.apache.org > *Subject:* Re: PermGen leak in JBoss**** > > ** ** > > ** ** > > ** ** > > On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson < > HAbelson@netcentricinc.com> wrote:**** > > Hi, I am a colleague of Michael's, I can provide more info > > The following are the 5 threads that are not being cleaned up on an > undeploy. It's hard to get more stack info than this. > > Daemon Thread [FileWatchdog] (Running) > Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] > (Running) > Daemon Thread [MSC service thread 1-5-EventThread] (Running) > Daemon Thread [Thrift Connection Pool Checker] (Running) > Daemon Thread [FileWatchdog] (Running) > > Using Plumbr, the report states that the SendThread and EventThread above > both contain an instance of org.apache.zookeeper.ClientCnxn**** > > ** ** > > ** ** > > I tried this on 1.4 and then ran jstack using following program. **** > > ** ** > > ** ** > > public class Test {**** > > public static void main(String[] args) throws Exception {**** > > ZooKeeperInstance instance =3D new ZooKeeperInstance("test14", > "localhost");**** > > Connector connector =3D instance.getConnector("root", "secret");**** > > **** > > UtilWaitThread.sleep(600000);**** > > ** ** > > }**** > > }**** > > ** ** > > I see the following threads after filtering out java threads and the > thread calling main. I am running this code in a different way. I used > the accumulo script to execute it, which uses Accumulo start, which > introduces the AccumuloFilesystemAlterationMonitor thread. I am seeing > three of the threads you see, I am not seeing the FileWatchDog threads.**= * > * > > ** ** > > "Thrift Connection Pool Checker" daemon prio=3D10 tid=3D0x00007f861839f80= 0 > nid=3D0x61f0 waiting on condition [0x00007f8652a62000]**** > > java.lang.Thread.State: TIMED_WAITING (sleeping)**** > > at java.lang.Thread.sleep(Native Method)**** > > at > org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(Thrif= tTransportPool.java:129) > **** > > at java.lang.Thread.run(Thread.java:662)**** > > ** ** > > "Test-EventThread" daemon prio=3D10 tid=3D0x00007f8618119000 nid=3D0x61ee > waiting on condition [0x00007f8652d02000]**** > > java.lang.Thread.State: WAITING (parking)**** > > at sun.misc.Unsafe.park(Native Method)**** > > - parking to wait for <0x00000000bb402270> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)***= * > > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)**** > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awa= it(AbstractQueuedSynchronizer.java:1987) > **** > > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:39= 9) > **** > > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)**** > > ** ** > > "Test-SendThread(localhost:2181)" daemon prio=3D10 tid=3D0x00007f86181030= 00 > nid=3D0x61ed runnable [0x00007f8652e03000]**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)**** > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)**** > > at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)**** > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)**** > > - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)**** > > - locked <0x00000000bb3e0e78> (a > java.util.Collections$UnmodifiableSet)**** > > - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl= ) > **** > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)**** > > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)**** > > ** ** > > "org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonito= r" > daemon prio=3D10 tid=3D0x00007f86a4387000 nid=3D0x61e9 waiting on conditi= on > [0x00007f865367c000]**** > > java.lang.Thread.State: TIMED_WAITING (sleeping)**** > > at java.lang.Thread.sleep(Native Method)**** > > at > org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor= .run(AccumuloFilesystemAlterationMonitor.java:127) > **** > > at java.lang.Thread.run(Thread.java:662)**** > > ** ** > > **** > > Should probably open a ticket about this issue. We could probably do > something about the Thrift Connection Pool Checker threads. I am not su= re > if the zookeeper threads are an issue w/ zookeeper or an issue w/ how > accumulo is using zookeeper.**** > > ** ** > > > -Heath**** > > > -----Original Message----- > From: Michael Giordano > Sent: Wednesday, April 24, 2013 9:48 AM > To: user@accumulo.apache.org > Subject: PermGen leak in JBoss > > Under version 1.3.7 we are using the following code to initialize a > cloudbase connection during initialization of our web app: > > ZooKeeperInstance instance =3D new > ZooKeeperInstance(instanceName, zooKeepers); > connector =3D instance.getConnector(userId, > password.getBytes()); > > The problem is that under the hood, this call creates several (5) threads > that are not cleaned up when the app is undeployed in JBoss. This is > occurring without performing any scans or interacting with cloudbase in a= ny > other way. After relatively few redeploys of the app, the PermGen Space i= s > OOM. > > I can't find any reference in the cloudbase API akin to a close() method > for the Connector object. This is a classloader leak effecting any webapp > that is accessing cloudbase directly. The result of this leak is not simp= ly > orphaned threads, but thousands of classes not gc'd because the classload= er > itself can't be gc'd. This is what is filling up PermGen. > > Has anyone discovered this particular issue? Has anyone discovered a > solution? > > Thanks, > Mike G. > > NetCentric Technology, Inc. > 3349 Route 138, Building A > Wall, NJ 07719 > Phone: 732-544-0888 > > **** > > ** ** > --bcaec50409ae6c261904dbd25cd7 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable



On Fri, May 3, 2013 at 11:43 AM, Michael Giordano <MGior= dano@netcentricinc.com> wrote:

Sorry for resurrecting this =91old=92 thread but I ne= ed to open a ticket for this.


<= /div>
=A0

=A0

How do I do that ?

=A0

Thanks,

Mike G.

=A0

From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, April 24, 2013 12:23 PM


To: us= er@accumulo.apache.org
Subject: Re: PermGen leak in JBoss

=A0

=A0

=A0

On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <HAbelson@netcentri= cinc.com> wrote:

Hi, I am a colleague of Michael's, I can provide more inf= o

The following are the 5 threads that are not being cleaned up on an undeplo= y. It's hard to get more stack info than this.

Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Runnin= g)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)

Using Plumbr, the report states that the SendThread and EventThread above b= oth contain an instance of org.apache.zookeeper.ClientCnxn

=A0

=A0

I tried this on 1.4 and then ran jstack using following progr= am. =A0

=A0

=A0

public class Test {

=A0 public static void main(String[] args) throws Exception {=

=A0 =A0 ZooKeeperInstance instance =3D new ZooKeeperInstance(= "test14", "localhost");

=A0 =A0 Connector connector =3D instance.getConnector("r= oot", "secret");

=A0 =A0=A0

=A0 =A0 UtilWaitThread.sleep(600000);

=A0

=A0 }

}

=A0

I see the following threads after filtering out java threads = and the thread calling main. =A0I am running this code in a different way. = =A0I used the accumulo script to execute it, which uses Accumulo start, whi= ch introduces the=A0AccumuloFilesystemAlterationMonitor thread. =A0I am seeing three of the threads you see, I am not seeing the F= ileWatchDog threads.

=A0

"Thrift Connection Pool Checker" daemon prio=3D10 t= id=3D0x00007f861839f800 nid=3D0x61f0 waiting on condition [0x00007f8652a620= 00]

=A0 =A0java.lang.Thread.State: TIMED_WAITING (sleeping)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.sleep(N= ative Method)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.accumulo.core= .client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thr= ead.java:662)

=A0

"Test-EventThread" daemon prio=3D10 tid=3D0x00007f8= 618119000 nid=3D0x61ee waiting on condition [0x00007f8652d02000]<= /u>

=A0 =A0java.lang.Thread.State: WAITING (parking)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.misc.Unsafe.park(Nat= ive Method)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - parking to wait for =A0&l= t;0x00000000bb402270> (a java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.loc= ks.LockSupport.park(LockSupport.java:156)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.loc= ks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchroni= zer.java:1987)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.Lin= kedBlockingQueue.take(LinkedBlockingQueue.java:399)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.zookeeper.Cli= entCnxn$EventThread.run(ClientCnxn.java:498)

=A0

"Test-SendThread(localhost:2181)" daemon prio=3D10 = tid=3D0x00007f8618103000 nid=3D0x61ed runnable [0x00007f8652e03000]<= u>

=A0 =A0java.lang.Thread.State: RUNNABLE

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.nio.ch.EPollArrayWra= pper.epollWait(Native Method)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.nio.ch.EPollArrayWra= pper.poll(EPollArrayWrapper.java:210)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.nio.ch.EPollSelector= Impl.doSelect(EPollSelectorImpl.java:65)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.nio.ch.SelectorImpl.= lockAndDoSelect(SelectorImpl.java:69)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - locked <0x00000000bb3e= 0e88> (a sun.nio.ch.Util$2)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - locked <0x00000000bb3e= 0e78> (a java.util.Collections$UnmodifiableSet)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - locked <0x00000000bb3e= 0aa8> (a sun.nio.ch.EPollSelectorImpl)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at sun.nio.ch.SelectorImpl.= select(SelectorImpl.java:80)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.zookeeper.Cli= entCnxn$SendThread.run(ClientCnxn.java:1131)

=A0

"org.apache.accumulo.start.classloader.AccumuloFilesyste= mAlterationMonitor" daemon prio=3D10 tid=3D0x00007f86a4387000 nid=3D0x= 61e9 waiting on condition [0x00007f865367c000]

=A0 =A0java.lang.Thread.State: TIMED_WAITING (sleeping)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.sleep(N= ative Method)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.accumulo.star= t.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlt= erationMonitor.java:127)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thr= ead.java:662)

=A0

=A0

Should probably open a ticket about this issue. =A0We could p= robably do something about the=A0Thrift Connection Pool Checker threads. = =A0 I am not sure if the zookeeper threads are an issue w/ zookeeper or an = issue w/ how accumulo is using zookeeper.

=A0


-Heath


-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accu= mulo.apache.org
Subject: PermGen leak in JBoss

Under version 1.3.7 we are using the following code to initialize a cloudba= se connection during initialization of our web app:

=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ZooKeeperInstance instance = =3D new ZooKeeperInstance(instanceName, zooKeepers);
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 connector =3D instance.getC= onnector(userId, password.getBytes());

The problem is that under the hood, this call creates several (5) threads t= hat are not cleaned up when the app is undeployed in JBoss. This is occurri= ng without performing any scans or interacting with cloudbase in any other = way. After relatively few redeploys of the app, the PermGen Space is OOM.

I can't find any reference in the cloudbase API akin to a close() metho= d for the Connector object. This is a classloader leak effecting any webapp= that is accessing cloudbase directly. The result of this leak is not simpl= y orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd.= This is what is filling up PermGen.

Has anyone discovered this particular issue? Has anyone discovered a soluti= on?

Thanks,
Mike G.

NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ =A007719
Phone: 732-544-0888

=A0


--bcaec50409ae6c261904dbd25cd7--