tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogesh Patel <yogesh.r.pa...@highq.com>
Subject Re: Regarding StuckThreadDetectionValve
Date Mon, 05 Oct 2015 10:42:00 GMT
Hi , let me try sending image using attachment , if still its not viewable
then i will find another way.

On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:

> On 05/10/2015 11:28, Yogesh Patel wrote:
> > Hi Thomas ,
> >                   Please see this image ...have a look at Time column
>
> The list strips images. If you really want us to look at the image (not
> that I think it will be remotely relevant) upload it somewhere and post
> the URL (make sure it is publicly accessible and doe snot require any
> form of registration to view it).
>
> Mark
>
>
> >
> >
> > ‚Äč
> >
> > On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
> > <mailto:markt@apache.org>> wrote:
> >
> >     On 05/10/2015 10:09, Yogesh Patel wrote:
> >     > Hi Thomas,
> >     >
> >     > Connector configuration is like :
> >     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" />
> >
> >     That means you are using the BIO AJP connector.
> >
> >     You don't have a problem with long running requests.
> >
> >     You have a problem with thread starvation.
> >
> >     AJP uses persistent connections.
> >
> >     BIO requires one thread per connection, regardless of whether or nor
> >     that connection is processing a request or waiting for the next one.
> >
> >     httpd will (eventually, assuming mostly default config) create one
> AJP
> >     connection for each httpd thread. If you have more httpd threads than
> >     Tomcat threads you will eventually reach the point where httpd can't
> >     create a Tomcat thread it requires and at that point it will appear
> >     to hang.
> >
> >     There are several possible fixes:
> >     a) disable connection re-use in httpd for AJP connections
> >     b) use the NIO AJP connector in Tomcat
> >     c) increase maxThreads in Tomcat so it is > max threads in httpd
> >
> >     For more explanation read this:
> >
> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >     <
> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >
> >
> >     from slide 29 to 33
> >
> >     Mark
> >
> >
> >
> >     >
> >     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
> >     <mailto:markt@apache.org>> wrote:
> >     >
> >     >> On 05/10/2015 09:07, Yogesh Patel wrote:
> >     >>> Thanks Mark Thomas ,
> >     >>>
> >     >>>    Our application is access by Apache TO Tomcat using AJP
> >     Connector.
> >     >>
> >     >> OK. That answers one of the questions I asked. However, you
> haven't
> >     >> provided the connector configuration.
> >     >>
> >     >>>     Problem :
> >     >>>     Our application was mostly hanged,after looking at tomcat
> >     manager it
> >     >>> shown there are so many long running threads shown.
> >     >>
> >     >> The Tomcat Manager app does not show long running threads. It
> >     does show
> >     >> you how many requests are busy but again, a busy thread is not
> >     >> necessarily processing a request.
> >     >>
> >     >>>     We want to recognize why so many threads are running since
> >     long time.
> >     >>
> >     >> Threads are always "running". The key question is are those
> threads
> >     >> 'idle' or 'busy'. An idle thread is waiting to be allocated work.
> >     A busy
> >     >> thread has been allocated work but the manager application can't
> >     >> distinguish if that work is 'wait for a request to process' or
> >     >> 'processing a request'.
> >     >>
> >     >>>     We want to detect such thread and want to kill these stucked
> >     thread.
> >     >>
> >     >> You continue to make the (increasingly unlikely) assumption that
> 200
> >     >> busy threads mean you have 200 threads processing long running
> >     requests.
> >     >> Had you provided the connector configuration I asked for (by that
> >     I mean
> >     >> the <Connector ... /> elements in server.xml) then I'd be in
a
> better
> >     >> position to tell you what is wrong.
> >     >>
> >     >> If you do have 200 long running requests then the
> >     >> StuckThreadDetectionValve is how you detect them. That fact that
> that
> >     >> valve is not reporting any long running requests should be a big
> clue
> >     >> that your assumptions about what is going on are not correct.
> >     >>
> >     >> If, and it is a big if, you did have 200 concurrent long running
> >     >> requests then there is no guaranteed way to stop them. If your
> >     >> application checks for interrupts (most don't) then the
> >     >> StuckThreadDetectionValve can interrupt them which should
> >     terminate the
> >     >> processing of that request but the time it would take to code the
> >     >> application to handle that properly would normally be better spent
> >     >> fixing the root cause of the long running request.
> >     >>
> >     >> Mark
> >     >>
> >     >>>
> >     >>>
> >     >>>
> >     >>>
> >     >>>
> >     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
> >     <mailto:markt@apache.org>> wrote:
> >     >>>
> >     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
> >     >>>>>         We are facing issues with long running thread in
> >     tomcat . we
> >     >> are
> >     >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy threads
> >     : 200,
> >     >>>>> application  gets stucked due to these long running threads.
> >     >>>>
> >     >>>> What makes you think you have issues with long running threads?
> >     A busy
> >     >>>> thread isn't necessarily processing a request. Configuration
> errors
> >     >>>> leading to thread starvation are a more typical cause of the
> >     symptom you
> >     >>>> describe rather than long running threads in the application.
> >     >>>>
> >     >>>>>          We implemented StuckThreadDetectionValve in
> >     server.xml( <Valve
> >     >>>>>
> >     >>>>>
> >      className="org.apache.catalina.valves.StuckThreadDetectionValve"
> >     >>>>>
> >     >>>>>     threshold="60" />), but it  could not help out.
> >     >>>>
> >     >>>> Why not? Again, this suggests that long running threads aren't
> the
> >     >> problem.
> >     >>>>
> >     >>>>>        So we implemented custom StuckThreadDetectionValve
by
> >     extending
> >     >>>>> StuckThreadDetectionValve from
> >     >>>>> catalina, it only goes to "constructor","initInternal",and
in
> >     >>>> "invoke"(when
> >     >>>>> action fires), it does not go to function
> >     "getStuckThreadNames()" even
> >     >>>>> after threshold time.How to achieve the same.Is there any
way
> >     to detect
> >     >>>>> stucked thread and kill them?
> >     >>>>
> >     >>>> First of all your invoke() method fails to call super.invoke()
> >     so the
> >     >>>> Valve is never going to do anything.
> >     >>>>
> >     >>>> Secondly, if the original valve didn't work adding a bunch
of
> >     >>>> System.out.println() lines isn't going to magically make it
> work.
> >     >>>>
> >     >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat
> >     code so it
> >     >>>> is no surprise that you never see a call to that method. That
> >     method is
> >     >>>> intended for use via JMX.
> >     >>>>
> >     >>>> I suggest you start again and tell us more about the problem
> >     you are
> >     >>>> trying to solve (lack of response) and your configuration.
In
> >     particular
> >     >>>> we need to know your connector configuration and how users
> >     access the
> >     >>>> application. Do they connect directly to Tomcat or do they
go
> via a
> >     >>>> reverse proxy?
> >     >>>>
> >     >>>> Mark
> >     >>>>
> >     >>>>
> >     >>>>>
> >     >>>>>   My custom Valve is like following :
> >     >>>>>
> >     >>>>> public class StuckThreadDetection extends
> >     StuckThreadDetectionValve
> >     >>>>> {
> >     >>>>> StuckThreadDetection stuckThreadDetection;
> >     >>>>>
> >     >>>>> public StuckThreadDetection()
> >     >>>>> {
> >     >>>>> System.out.println("in StuckThreadDetection constructor");
> >     >>>>>
> >     >>>>> }
> >     >>>>>
> >     >>>>> public void invoke(Request request, Response response)
throws
> >     >>>> IOException,
> >     >>>>> ServletException
> >     >>>>> {
> >     >>>>> System.out.println("in invoke...");
> >     >>>>>
> >     >>>>> getNext().invoke(request, response);
> >     >>>>> }
> >     >>>>>
> >     >>>>> @Override
> >     >>>>> protected void initInternal() throws LifecycleException
> >     >>>>> {
> >     >>>>> System.out.println("In initInternal");
> >     >>>>> super.initInternal();
> >     >>>>> }
> >     >>>>>
> >     >>>>> @Override
> >     >>>>> public String[] getStuckThreadNames()
> >     >>>>> {
> >     >>>>> System.out.println("in getStuckThreadNames...");
> >     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
> >     >>>>>
> >     >>>>> for (String threadName : listStuckedThread)
> >     >>>>> {
> >     >>>>> System.out.println(threadName);
> >     >>>>> }
> >     >>>>>
> >     >>>>> return listStuckedThread;
> >     >>>>>
> >     >>>>> }
> >     >>>>>
> >     >>>>> @Override
> >     >>>>> public String getInfo()
> >     >>>>> {
> >     >>>>> System.out.println("In getInfo");
> >     >>>>> return super.getInfo();
> >     >>>>> }
> >     >>>>> }
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>
> >     >>>>
> >     >>>>
> >     ---------------------------------------------------------------------
> >     >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >     <mailto:users-unsubscribe@tomcat.apache.org>
> >     >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >     <mailto:users-help@tomcat.apache.org>
> >     >>>>
> >     >>>>
> >     >>>
> >     >>>
> >     >>
> >     >>
> >     >>
> ---------------------------------------------------------------------
> >     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >     <mailto:users-unsubscribe@tomcat.apache.org>
> >     >> For additional commands, e-mail: users-help@tomcat.apache.org
> >     <mailto:users-help@tomcat.apache.org>
> >     >>
> >     >>
> >     >
> >     >
> >
> >
> >     ---------------------------------------------------------------------
> >     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >     <mailto:users-unsubscribe@tomcat.apache.org>
> >     For additional commands, e-mail: users-help@tomcat.apache.org
> >     <mailto:users-help@tomcat.apache.org>
> >
> >
> >
> >
> > --
> > /Thanks & Regards,/
> > / /
> > / Yogesh Patel/
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


-- 
*Thanks & Regards,*

* Yogesh Patel*

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message