Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
MIME-Version: 1.0
In-Reply-To: <5612D160.7050203@apache.org>
References: 
 <CALpj=OYDqx7H-FcAhJbY6pOHMTQj-AT5GJunbtPj7hDiE_h79A@mail.gmail.com>
	<5612298F.9020701@apache.org>
	<CALpj=OZuC3MS0gff69rd6B3RerNGOhHcwqtLM6LmOh1Z4C4mXg@mail.gmail.com>
	<56123936.5090009@apache.org>
	<CALpj=OZfjPZ8VWZ94qFTmvCiUvGeBkvpBa6h8fY47OHiMYfu9w@mail.gmail.com>
	<561240E4.6090306@apache.org>
	<CALpj=ObUm9jvAfnxzKU=cmhNTjS_Z-e_JmWW+=7RSBuz00SBew@mail.gmail.com>
	<561252A1.3060106@apache.org>
	<CALpj=OaO=uzOigQafYaiZNTcuQ11kcXRGxU=pWhZ6WoXsnt7yA@mail.gmail.com>
	<CALpj=OZPmxHiiJyiNLHdfMtU-wBUUaUv=CJTWgpkqaBK696vsg@mail.gmail.com>
	<56128DBA.1030403@apache.org>
	<CALpj=ObeQtHzn2VAajm3kgeTo-ppztJnCCmX-+WcG7rNzBF4SA@mail.gmail.com>
	<5612D160.7050203@apache.org>
Date: Tue, 6 Oct 2015 11:36:27 +0530
Message-ID: 
 <CALpj=OY7TtkEMSuSpOWPsHQcd2qk_T4HG9sjgXP38RtmVMWw7Q@mail.gmail.com>
Subject: Re: Regarding StuckThreadDetectionValve
From: Yogesh Patel <yogesh.r.patel@highq.com>
To: Tomcat Users List <users@tomcat.apache.org>
Content-Type: multipart/alternative; boundary=047d7bd6ab642dcff50521696ecb

--047d7bd6ab642dcff50521696ecb
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks Thomas,

            I just wanted to know in tomcat is there any way to detect such
long running thread and kill them.

On 6 October 2015 at 01:07, Mark Thomas <markt@apache.org> wrote:

> On 05/10/2015 16:00, Yogesh Patel wrote:
> > Other 199 process are also for /solr430/update?wt=3Dxml&version=3D2.2
> HTTP/1.1
> > only
>
> That is surprising. That could be a Solr issue or a Tomcat issue. Best
> to take three thread dumps each 10 to 30 seconds apart and then take a
> look to see what is holding up those threads. Best guess is a deadlock
> or a database issue.
>
> (You still have the potential for thread starvation that will come back
> to bite you later after you fix this issue unless you fix it)
>
> Mark
>
> >
> > On 5 October 2015 at 20:18, Mark Thomas <markt@apache.org> wrote:
> >
> >> On 05/10/2015 12:58, Yogesh Patel wrote:
> >>> Hi Mark Thomas,
> >>>
> >>> in image it shows Tomcat Manager screen:
> >>> Under "ajp-apr-10003" Section in tomcat manager:
> >>> Max threads: 200 Current thread count:200 Current thread busy :200
> Keeped
> >>> alive socket count :0
> >>>
> >>> *Stage   Time              BSent    BRecv          Client         VHo=
st
> >>>  Request*
> >>> S          2884466 ms    0 KB     184063 KB     machineip     host
> >>  POST
> >>> /solr430/update?wt=3Dxml&version=3D2.2 HTTP/1.1
> >>
> >> And the other 199 processors?
> >>
> >> Mark
> >>
> >>
> >>>
> >>> ..........
> >>> ............
> >>>
> >>>
> >>> On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.patel@highq.com>
> >> wrote:
> >>>
> >>>> Hi , let me try sending image using attachment , if still its not
> >> viewable
> >>>> then i will find another way.
> >>>>
> >>>> On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:
> >>>>
> >>>>> On 05/10/2015 11:28, Yogesh Patel wrote:
> >>>>>> Hi Thomas ,
> >>>>>>                   Please see this image ...have a look at Time
> column
> >>>>>
> >>>>> The list strips images. If you really want us to look at the image
> (not
> >>>>> that I think it will be remotely relevant) upload it somewhere and
> post
> >>>>> the URL (make sure it is publicly accessible and doe snot require a=
ny
> >>>>> form of registration to view it).
> >>>>>
> >>>>> Mark
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> =E2=80=8B
> >>>>>>
> >>>>>> On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
> >>>>>> <mailto:markt@apache.org>> wrote:
> >>>>>>
> >>>>>>     On 05/10/2015 10:09, Yogesh Patel wrote:
> >>>>>>     > Hi Thomas,
> >>>>>>     >
> >>>>>>     > Connector configuration is like :
> >>>>>>     > <Connector port=3D"10004" protocol=3D"AJP/1.3" redirectPort=
=3D"8443"
> >> />
> >>>>>>
> >>>>>>     That means you are using the BIO AJP connector.
> >>>>>>
> >>>>>>     You don't have a problem with long running requests.
> >>>>>>
> >>>>>>     You have a problem with thread starvation.
> >>>>>>
> >>>>>>     AJP uses persistent connections.
> >>>>>>
> >>>>>>     BIO requires one thread per connection, regardless of whether =
or
> >> nor
> >>>>>>     that connection is processing a request or waiting for the nex=
t
> >> one.
> >>>>>>
> >>>>>>     httpd will (eventually, assuming mostly default config) create
> one
> >>>>> AJP
> >>>>>>     connection for each httpd thread. If you have more httpd threa=
ds
> >>>>> than
> >>>>>>     Tomcat threads you will eventually reach the point where httpd
> >> can't
> >>>>>>     create a Tomcat thread it requires and at that point it will
> >> appear
> >>>>>>     to hang.
> >>>>>>
> >>>>>>     There are several possible fixes:
> >>>>>>     a) disable connection re-use in httpd for AJP connections
> >>>>>>     b) use the NIO AJP connector in Tomcat
> >>>>>>     c) increase maxThreads in Tomcat so it is > max threads in htt=
pd
> >>>>>>
> >>>>>>     For more explanation read this:
> >>>>>>
> >>>>>
> >>
> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clusterin=
g-part-1-reverse-proxies.pdf
> >>>>>>     <
> >>>>>
> >>
> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-cluster=
ing-part-1-reverse-proxies.pdf
> >>>>>>
> >>>>>>
> >>>>>>     from slide 29 to 33
> >>>>>>
> >>>>>>     Mark
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>     >
> >>>>>>     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
> >>>>>>     <mailto:markt@apache.org>> wrote:
> >>>>>>     >
> >>>>>>     >> On 05/10/2015 09:07, Yogesh Patel wrote:
> >>>>>>     >>> Thanks Mark Thomas ,
> >>>>>>     >>>
> >>>>>>     >>>    Our application is access by Apache TO Tomcat using AJP
> >>>>>>     Connector.
> >>>>>>     >>
> >>>>>>     >> OK. That answers one of the questions I asked. However, you
> >>>>> haven't
> >>>>>>     >> provided the connector configuration.
> >>>>>>     >>
> >>>>>>     >>>     Problem :
> >>>>>>     >>>     Our application was mostly hanged,after looking at
> tomcat
> >>>>>>     manager it
> >>>>>>     >>> shown there are so many long running threads shown.
> >>>>>>     >>
> >>>>>>     >> The Tomcat Manager app does not show long running threads. =
It
> >>>>>>     does show
> >>>>>>     >> you how many requests are busy but again, a busy thread is
> not
> >>>>>>     >> necessarily processing a request.
> >>>>>>     >>
> >>>>>>     >>>     We want to recognize why so many threads are running
> since
> >>>>>>     long time.
> >>>>>>     >>
> >>>>>>     >> Threads are always "running". The key question is are those
> >>>>> threads
> >>>>>>     >> 'idle' or 'busy'. An idle thread is waiting to be allocated
> >> work.
> >>>>>>     A busy
> >>>>>>     >> thread has been allocated work but the manager application
> >> can't
> >>>>>>     >> distinguish if that work is 'wait for a request to process'
> or
> >>>>>>     >> 'processing a request'.
> >>>>>>     >>
> >>>>>>     >>>     We want to detect such thread and want to kill these
> >> stucked
> >>>>>>     thread.
> >>>>>>     >>
> >>>>>>     >> You continue to make the (increasingly unlikely) assumption
> >> that
> >>>>> 200
> >>>>>>     >> busy threads mean you have 200 threads processing long
> running
> >>>>>>     requests.
> >>>>>>     >> Had you provided the connector configuration I asked for (b=
y
> >> that
> >>>>>>     I mean
> >>>>>>     >> the <Connector ... /> elements in server.xml) then I'd be i=
n
> a
> >>>>> better
> >>>>>>     >> position to tell you what is wrong.
> >>>>>>     >>
> >>>>>>     >> If you do have 200 long running requests then the
> >>>>>>     >> StuckThreadDetectionValve is how you detect them. That fact
> >> that
> >>>>> that
> >>>>>>     >> valve is not reporting any long running requests should be =
a
> >> big
> >>>>> clue
> >>>>>>     >> that your assumptions about what is going on are not correc=
t.
> >>>>>>     >>
> >>>>>>     >> If, and it is a big if, you did have 200 concurrent long
> >> running
> >>>>>>     >> requests then there is no guaranteed way to stop them. If
> your
> >>>>>>     >> application checks for interrupts (most don't) then the
> >>>>>>     >> StuckThreadDetectionValve can interrupt them which should
> >>>>>>     terminate the
> >>>>>>     >> processing of that request but the time it would take to co=
de
> >> the
> >>>>>>     >> application to handle that properly would normally be bette=
r
> >>>>> spent
> >>>>>>     >> fixing the root cause of the long running request.
> >>>>>>     >>
> >>>>>>     >> Mark
> >>>>>>     >>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
> >>>>>>     <mailto:markt@apache.org>> wrote:
> >>>>>>     >>>
> >>>>>>     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
> >>>>>>     >>>>>         We are facing issues with long running thread in
> >>>>>>     tomcat . we
> >>>>>>     >> are
> >>>>>>     >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy
> >> threads
> >>>>>>     : 200,
> >>>>>>     >>>>> application  gets stucked due to these long running
> threads.
> >>>>>>     >>>>
> >>>>>>     >>>> What makes you think you have issues with long running
> >> threads?
> >>>>>>     A busy
> >>>>>>     >>>> thread isn't necessarily processing a request.
> Configuration
> >>>>> errors
> >>>>>>     >>>> leading to thread starvation are a more typical cause of
> the
> >>>>>>     symptom you
> >>>>>>     >>>> describe rather than long running threads in the
> application.
> >>>>>>     >>>>
> >>>>>>     >>>>>          We implemented StuckThreadDetectionValve in
> >>>>>>     server.xml( <Valve
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>
> className=3D"org.apache.catalina.valves.StuckThreadDetectionValve"
> >>>>>>     >>>>>
> >>>>>>     >>>>>     threshold=3D"60" />), but it  could not help out.
> >>>>>>     >>>>
> >>>>>>     >>>> Why not? Again, this suggests that long running threads
> >> aren't
> >>>>> the
> >>>>>>     >> problem.
> >>>>>>     >>>>
> >>>>>>     >>>>>        So we implemented custom StuckThreadDetectionValv=
e
> by
> >>>>>>     extending
> >>>>>>     >>>>> StuckThreadDetectionValve from
> >>>>>>     >>>>> catalina, it only goes to "constructor","initInternal",a=
nd
> >> in
> >>>>>>     >>>> "invoke"(when
> >>>>>>     >>>>> action fires), it does not go to function
> >>>>>>     "getStuckThreadNames()" even
> >>>>>>     >>>>> after threshold time.How to achieve the same.Is there an=
y
> >> way
> >>>>>>     to detect
> >>>>>>     >>>>> stucked thread and kill them?
> >>>>>>     >>>>
> >>>>>>     >>>> First of all your invoke() method fails to call
> >> super.invoke()
> >>>>>>     so the
> >>>>>>     >>>> Valve is never going to do anything.
> >>>>>>     >>>>
> >>>>>>     >>>> Secondly, if the original valve didn't work adding a bunc=
h
> of
> >>>>>>     >>>> System.out.println() lines isn't going to magically make =
it
> >>>>> work.
> >>>>>>     >>>>
> >>>>>>     >>>> Thirdly, getStuckThreadNames() is never called by any
> Tomcat
> >>>>>>     code so it
> >>>>>>     >>>> is no surprise that you never see a call to that method.
> That
> >>>>>>     method is
> >>>>>>     >>>> intended for use via JMX.
> >>>>>>     >>>>
> >>>>>>     >>>> I suggest you start again and tell us more about the
> problem
> >>>>>>     you are
> >>>>>>     >>>> trying to solve (lack of response) and your configuration=
.
> In
> >>>>>>     particular
> >>>>>>     >>>> we need to know your connector configuration and how user=
s
> >>>>>>     access the
> >>>>>>     >>>> application. Do they connect directly to Tomcat or do the=
y
> go
> >>>>> via a
> >>>>>>     >>>> reverse proxy?
> >>>>>>     >>>>
> >>>>>>     >>>> Mark
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>>   My custom Valve is like following :
> >>>>>>     >>>>>
> >>>>>>     >>>>> public class StuckThreadDetection extends
> >>>>>>     StuckThreadDetectionValve
> >>>>>>     >>>>> {
> >>>>>>     >>>>> StuckThreadDetection stuckThreadDetection;
> >>>>>>     >>>>>
> >>>>>>     >>>>> public StuckThreadDetection()
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("in StuckThreadDetection constructor"=
);
> >>>>>>     >>>>>
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> public void invoke(Request request, Response response)
> >> throws
> >>>>>>     >>>> IOException,
> >>>>>>     >>>>> ServletException
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("in invoke...");
> >>>>>>     >>>>>
> >>>>>>     >>>>> getNext().invoke(request, response);
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> @Override
> >>>>>>     >>>>> protected void initInternal() throws LifecycleException
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("In initInternal");
> >>>>>>     >>>>> super.initInternal();
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> @Override
> >>>>>>     >>>>> public String[] getStuckThreadNames()
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("in getStuckThreadNames...");
> >>>>>>     >>>>> String[] listStuckedThread =3D this.getStuckThreadNames(=
);
> >>>>>>     >>>>>
> >>>>>>     >>>>> for (String threadName : listStuckedThread)
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println(threadName);
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> return listStuckedThread;
> >>>>>>     >>>>>
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> @Override
> >>>>>>     >>>>> public String getInfo()
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("In getInfo");
> >>>>>>     >>>>> return super.getInfo();
> >>>>>>     >>>>> }
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>>>     >>>> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> >>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>>>     >>>> For additional commands, e-mail:
> >> users-help@tomcat.apache.org
> >>>>>>     <mailto:users-help@tomcat.apache.org>
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>
> >>>>>>     >>
> >>>>>>     >>
> >>>>> -------------------------------------------------------------------=
--
> >>>>>>     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>>>     >> For additional commands, e-mail:
> users-help@tomcat.apache.org
> >>>>>>     <mailto:users-help@tomcat.apache.org>
> >>>>>>     >>
> >>>>>>     >>
> >>>>>>     >
> >>>>>>     >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>>>     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>>>     For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>     <mailto:users-help@tomcat.apache.org>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> /Thanks & Regards,/
> >>>>>> / /
> >>>>>> / Yogesh Patel/
> >>>>>
> >>>>>
> >>>>> -------------------------------------------------------------------=
--
> >>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Thanks & Regards,*
> >>>>
> >>>> * Yogesh Patel*
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


--=20
*Thanks & Regards,*

* Yogesh Patel*

--047d7bd6ab642dcff50521696ecb--