Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 209.85.223.171 as permitted sender)
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: 
 <CABNOTEW2Hfu0pZfNWO-Aui2Xiz8uVyqmvrkAs_519eWARwZOiw@mail.gmail.com>
References: 
 <CABNOTEW-xz7yFO3S3jCQpponF8efbBnqT-KBhR4NCTF=EpkDgA@mail.gmail.com>
	<DA5848E6-2C17-40C6-ABC2-534A62905988@gmail.com>
	<CABNOTEWPOnj60zYhEz9Ns86X6rM1Ud21Gx-Opk4spzLBrh=agA@mail.gmail.com>
	<CALte62wqx3=9m7vi=4Vgw6XE=_i+ttD-JwYpm1a-p_RhG==5dg@mail.gmail.com>
	<CABNOTEXyvSZA-MpkJURYCPb1TeQvQ4o_d5ik6yc8Oiq=Cv91SA@mail.gmail.com>
	<CALte62wJa1j9-8HDvA0SVGRN9x5jckzXHxyO_2_-tQ6xume8fA@mail.gmail.com>
	<CABNOTEVqSMYmW5=oFSafp7Q8JZ9Fn6oFjHSyGbOtGQ+YK3r8SQ@mail.gmail.com>
	<CADcMMgE8UfAJ8SAs6AMAZ36kdcFoLNGcmVOkWRzX1YOPYfrasw@mail.gmail.com>
	<CABNOTEW2Hfu0pZfNWO-Aui2Xiz8uVyqmvrkAs_519eWARwZOiw@mail.gmail.com>
Date: Tue, 6 Jan 2015 12:31:04 -0800
Message-ID: 
 <CADcMMgFGm5WSxcbf-d=7Ct4gVapGM9_DjTq5DDB6h0vZS=fFGw@mail.gmail.com>
Subject: Re: Threads leaking from Apache tomcat application
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=bcaec510194da08203050c01afcc

--bcaec510194da08203050c01afcc
Content-Type: text/plain; charset=UTF-8

The threads that are sticking around are tomcat threads out of a tomcat
executor pool. IIRC, your server has high traffic.  The pool is running up
to 800 connections on occasion and taking a while to die back down?
Googling, seems like this issue comes up frequently enough. Try it
yourself.  If you can't figure something like putting a bound on the
executor, come back here and we'll try and help you out.

St.Ack

On Tue, Jan 6, 2015 at 12:10 PM, Serega Sheypak <serega.sheypak@gmail.com>
wrote:

> Hi, yes, it was me.
> I've followed advices, ZK connections on server side are stable.
> Here is current state of Tomcat:
> http://bigdatapath.com/wp-content/uploads/2015/01/002_jvisualvm_summary.png
> There are more than 800 threads and daemon threads.
>
> and the state of three ZK servers:
> http://bigdatapath.com/wp-content/uploads/2015/01/001_zk_server_state.png
>
> here is pastebin:
> http://pastebin.com/Cq8ppg08
>
> P.S.
> Looks like tomcat is running on OpenJDK 64-Bit Server VM.
> I'll ask to fix it, it should be Oracle 7 JDK
>
> 2015-01-06 20:43 GMT+03:00 Stack <stack@duboce.net>:
>
> > On Tue, Jan 6, 2015 at 4:52 AM, Serega Sheypak <serega.sheypak@gmail.com
> >
> > wrote:
> >
> > > yes, one of them (random) gets more connections than others.
> > >
> > > 9.3.1.1 Is OK.
> > > I have 1 HConnection for logical module per application and each
> > > ServletRequest gets it's own HTable. HTable closed each tme after
> > > ServletRequest is done. HConnection is never closed.
> > >
> > >
> > This is you, right: http://search-hadoop.com/m/DHED4lJSA32
> >
> > Then, we were leaking zk connections.  Is that fixed?
> >
> > Can you reproduce in the small?  By setting up your webapp deploy in test
> > bed and watching it for leaking?
> >
> > For this issue, can you post a thread dump in postbin or gist so can see?
> >
> > Can you post code too?
> >
> > St.Ack
> >
> >
> >
> > > 2015-01-05 21:22 GMT+03:00 Ted Yu <yuzhihong@gmail.com>:
> > >
> > > > In 022_zookeeper_metrics.png, server names are anonymized. Looks like
> > > only
> > > > one server got high number of connections.
> > > >
> > > > Have you seen 9.3.1.1 of http://hbase.apache.org/book.html#client ?
> > > >
> > > > Cheers
> > > >
> > > > On Mon, Jan 5, 2015 at 8:57 AM, Serega Sheypak <
> > serega.sheypak@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi, here is repost with images link
> > > > >
> > > > > Hi, I'm still trying to deal with apache tomcat web-app and hbase
> > HBase
> > > > > 0.98.6
> > > > > The root problem is that user threads constantly grows. I do get
> > > > thousands
> > > > > of live threads on tomcat instance. Then it dies of course.
> > > > >
> > > > > please see visualVM threads count dynamics
> > > > >
> > > >
> > >
> >
> http://bigdatapath.com/wp-content/uploads/2015/01/01_threads_count-grow.png
> > > > >
> > > > >
> > > > > Please see selected thread. It should be related to zookeeper
> > (because
> > > of
> > > > > thread-name suffix SendThread)
> > > > >
> > > > >
> > > >
> > >
> >
> http://bigdatapath.com/wp-content/uploads/2015/01/011_long_running_threads.png
> > > > >
> > > > > The threaddump for this thread is:
> > > > >
> > > > > "visit-thread-27799752116280271-EventThread" - Thread t@75
> > > > >    java.lang.Thread.State: WAITING
> > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > - parking to wait for <34671cea> (a
> > > > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> > > > >
> > > > >    Locked ownable synchronizers:
> > > > > - None
> > > > >
> > > > > Why does it live "forever"? I next 24 hours I would get ~1200 live
> > > > theads.
> > > > >
> > > > > "visit thread" does simple put/get by key, newrelic says it takes
> > 30-40
> > > > ms
> > > > > to respond.
> > > > > I just set a name for the thread inside servlet method.
> > > > >
> > > > > Here is CPU profiling result:
> > > > >
> > http://bigdatapath.com/wp-content/uploads/2015/01/03_cpu_prifling.png
> > > > >
> > > > > Here is zookeeper status:
> > > > >
> > > >
> > >
> >
> http://bigdatapath.com/wp-content/uploads/2015/01/022_zookeeper_metrics.png
> > > > >
> > > > > How can I debug and get root cause for long living threads? Looks
> > like
> > > I
> > > > > got threads leaking, but have no Idea why...
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2015-01-05 17:57 GMT+03:00 Ted Yu <yuzhihong@gmail.com>:
> > > > >
> > > > > > I used gmail.
> > > > > >
> > > > > > Please consider using third party site where you can upload
> images.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--bcaec510194da08203050c01afcc--