accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: PermGen leak in JBoss
Date Fri, 03 May 2013 15:54:06 GMT
On Fri, May 3, 2013 at 11:43 AM, Michael Giordano <
MGiordano@netcentricinc.com> wrote:

>  Sorry for resurrecting this ‘old’ thread but I need to open a ticket for
> this.
>

https://issues.apache.org/jira/browse/ACCUMULO


> ****
>
> ** **
>
> How do I do that ?****
>
> ** **
>
> Thanks,****
>
> Mike G.****
>
> ** **
>
> *From:* Keith Turner [mailto:keith@deenlo.com]
> *Sent:* Wednesday, April 24, 2013 12:23 PM
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: PermGen leak in JBoss****
>
> ** **
>
> ** **
>
> ** **
>
> On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <
> HAbelson@netcentricinc.com> wrote:****
>
> Hi, I am a colleague of Michael's, I can provide more info
>
> The following are the 5 threads that are not being cleaned up on an
> undeploy. It's hard to get more stack info than this.
>
> Daemon Thread [FileWatchdog] (Running)
> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
> (Running)
> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
> Daemon Thread [Thrift Connection Pool Checker] (Running)
> Daemon Thread [FileWatchdog] (Running)
>
> Using Plumbr, the report states that the SendThread and EventThread above
> both contain an instance of org.apache.zookeeper.ClientCnxn****
>
> ** **
>
> ** **
>
> I tried this on 1.4 and then ran jstack using following program.  ****
>
> ** **
>
> ** **
>
> public class Test {****
>
>   public static void main(String[] args) throws Exception {****
>
>     ZooKeeperInstance instance = new ZooKeeperInstance("test14",
> "localhost");****
>
>     Connector connector = instance.getConnector("root", "secret");****
>
>     ****
>
>     UtilWaitThread.sleep(600000);****
>
> ** **
>
>   }****
>
> }****
>
> ** **
>
> I see the following threads after filtering out java threads and the
> thread calling main.  I am running this code in a different way.  I used
> the accumulo script to execute it, which uses Accumulo start, which
> introduces the AccumuloFilesystemAlterationMonitor thread.  I am seeing
> three of the threads you see, I am not seeing the FileWatchDog threads.***
> *
>
> ** **
>
> "Thrift Connection Pool Checker" daemon prio=10 tid=0x00007f861839f800
> nid=0x61f0 waiting on condition [0x00007f8652a62000]****
>
>    java.lang.Thread.State: TIMED_WAITING (sleeping)****
>
>             at java.lang.Thread.sleep(Native Method)****
>
>             at
> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
> ****
>
>             at java.lang.Thread.run(Thread.java:662)****
>
> ** **
>
> "Test-EventThread" daemon prio=10 tid=0x00007f8618119000 nid=0x61ee
> waiting on condition [0x00007f8652d02000]****
>
>    java.lang.Thread.State: WAITING (parking)****
>
>             at sun.misc.Unsafe.park(Native Method)****
>
>             - parking to wait for  <0x00000000bb402270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
>             at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)****
>
>             at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> ****
>
>             at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> ****
>
>             at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)****
>
> ** **
>
> "Test-SendThread(localhost:2181)" daemon prio=10 tid=0x00007f8618103000
> nid=0x61ed runnable [0x00007f8652e03000]****
>
>    java.lang.Thread.State: RUNNABLE****
>
>             at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)****
>
>             at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)****
>
>             at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)****
>
>             at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)****
>
>             - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)****
>
>             - locked <0x00000000bb3e0e78> (a
> java.util.Collections$UnmodifiableSet)****
>
>             - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl)
> ****
>
>             at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)****
>
>             at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)****
>
> ** **
>
> "org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor"
> daemon prio=10 tid=0x00007f86a4387000 nid=0x61e9 waiting on condition
> [0x00007f865367c000]****
>
>    java.lang.Thread.State: TIMED_WAITING (sleeping)****
>
>             at java.lang.Thread.sleep(Native Method)****
>
>             at
> org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127)
> ****
>
>             at java.lang.Thread.run(Thread.java:662)****
>
> ** **
>
>  ****
>
> Should probably open a ticket about this issue.  We could probably do
> something about the Thrift Connection Pool Checker threads.   I am not sure
> if the zookeeper threads are an issue w/ zookeeper or an issue w/ how
> accumulo is using zookeeper.****
>
> ** **
>
>
> -Heath****
>
>
> -----Original Message-----
> From: Michael Giordano
> Sent: Wednesday, April 24, 2013 9:48 AM
> To: user@accumulo.apache.org
> Subject: PermGen leak in JBoss
>
> Under version 1.3.7 we are using the following code to initialize a
> cloudbase connection during initialization of our web app:
>
>                         ZooKeeperInstance instance = new
> ZooKeeperInstance(instanceName, zooKeepers);
>                         connector = instance.getConnector(userId,
> password.getBytes());
>
> The problem is that under the hood, this call creates several (5) threads
> that are not cleaned up when the app is undeployed in JBoss. This is
> occurring without performing any scans or interacting with cloudbase in any
> other way. After relatively few redeploys of the app, the PermGen Space is
> OOM.
>
> I can't find any reference in the cloudbase API akin to a close() method
> for the Connector object. This is a classloader leak effecting any webapp
> that is accessing cloudbase directly. The result of this leak is not simply
> orphaned threads, but thousands of classes not gc'd because the classloader
> itself can't be gc'd. This is what is filling up PermGen.
>
> Has anyone discovered this particular issue? Has anyone discovered a
> solution?
>
> Thanks,
> Mike G.
>
> NetCentric Technology, Inc.
> 3349 Route 138, Building A
> Wall, NJ  07719
> Phone: 732-544-0888
>
> ****
>
>  ** **
>

Mime
View raw message