tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogesh Patel <yogesh.r.pa...@highq.com>
Subject Re: Regarding StuckThreadDetectionValve
Date Mon, 05 Oct 2015 15:00:04 GMT
Other 199 process are also for /solr430/update?wt=xml&version=2.2 HTTP/1.1
only

On 5 October 2015 at 20:18, Mark Thomas <markt@apache.org> wrote:

> On 05/10/2015 12:58, Yogesh Patel wrote:
> > Hi Mark Thomas,
> >
> > in image it shows Tomcat Manager screen:
> > Under "ajp-apr-10003" Section in tomcat manager:
> > Max threads: 200 Current thread count:200 Current thread busy :200 Keeped
> > alive socket count :0
> >
> > *Stage   Time              BSent    BRecv          Client         VHost
> >  Request*
> > S          2884466 ms    0 KB     184063 KB     machineip     host
>  POST
> > /solr430/update?wt=xml&version=2.2 HTTP/1.1
>
> And the other 199 processors?
>
> Mark
>
>
> >
> > ..........
> > ............
> >
> >
> > On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.patel@highq.com>
> wrote:
> >
> >> Hi , let me try sending image using attachment , if still its not
> viewable
> >> then i will find another way.
> >>
> >> On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:
> >>
> >>> On 05/10/2015 11:28, Yogesh Patel wrote:
> >>>> Hi Thomas ,
> >>>>                   Please see this image ...have a look at Time column
> >>>
> >>> The list strips images. If you really want us to look at the image (not
> >>> that I think it will be remotely relevant) upload it somewhere and post
> >>> the URL (make sure it is publicly accessible and doe snot require any
> >>> form of registration to view it).
> >>>
> >>> Mark
> >>>
> >>>
> >>>>
> >>>>
> >>>> ‚Äč
> >>>>
> >>>> On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
> >>>> <mailto:markt@apache.org>> wrote:
> >>>>
> >>>>     On 05/10/2015 10:09, Yogesh Patel wrote:
> >>>>     > Hi Thomas,
> >>>>     >
> >>>>     > Connector configuration is like :
> >>>>     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443"
> />
> >>>>
> >>>>     That means you are using the BIO AJP connector.
> >>>>
> >>>>     You don't have a problem with long running requests.
> >>>>
> >>>>     You have a problem with thread starvation.
> >>>>
> >>>>     AJP uses persistent connections.
> >>>>
> >>>>     BIO requires one thread per connection, regardless of whether or
> nor
> >>>>     that connection is processing a request or waiting for the next
> one.
> >>>>
> >>>>     httpd will (eventually, assuming mostly default config) create one
> >>> AJP
> >>>>     connection for each httpd thread. If you have more httpd threads
> >>> than
> >>>>     Tomcat threads you will eventually reach the point where httpd
> can't
> >>>>     create a Tomcat thread it requires and at that point it will
> appear
> >>>>     to hang.
> >>>>
> >>>>     There are several possible fixes:
> >>>>     a) disable connection re-use in httpd for AJP connections
> >>>>     b) use the NIO AJP connector in Tomcat
> >>>>     c) increase maxThreads in Tomcat so it is > max threads in httpd
> >>>>
> >>>>     For more explanation read this:
> >>>>
> >>>
> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >>>>     <
> >>>
> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >>>>
> >>>>
> >>>>     from slide 29 to 33
> >>>>
> >>>>     Mark
> >>>>
> >>>>
> >>>>
> >>>>     >
> >>>>     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
> >>>>     <mailto:markt@apache.org>> wrote:
> >>>>     >
> >>>>     >> On 05/10/2015 09:07, Yogesh Patel wrote:
> >>>>     >>> Thanks Mark Thomas ,
> >>>>     >>>
> >>>>     >>>    Our application is access by Apache TO Tomcat using
AJP
> >>>>     Connector.
> >>>>     >>
> >>>>     >> OK. That answers one of the questions I asked. However,
you
> >>> haven't
> >>>>     >> provided the connector configuration.
> >>>>     >>
> >>>>     >>>     Problem :
> >>>>     >>>     Our application was mostly hanged,after looking
at tomcat
> >>>>     manager it
> >>>>     >>> shown there are so many long running threads shown.
> >>>>     >>
> >>>>     >> The Tomcat Manager app does not show long running threads.
It
> >>>>     does show
> >>>>     >> you how many requests are busy but again, a busy thread
is not
> >>>>     >> necessarily processing a request.
> >>>>     >>
> >>>>     >>>     We want to recognize why so many threads are running
since
> >>>>     long time.
> >>>>     >>
> >>>>     >> Threads are always "running". The key question is are those
> >>> threads
> >>>>     >> 'idle' or 'busy'. An idle thread is waiting to be allocated
> work.
> >>>>     A busy
> >>>>     >> thread has been allocated work but the manager application
> can't
> >>>>     >> distinguish if that work is 'wait for a request to process'
or
> >>>>     >> 'processing a request'.
> >>>>     >>
> >>>>     >>>     We want to detect such thread and want to kill
these
> stucked
> >>>>     thread.
> >>>>     >>
> >>>>     >> You continue to make the (increasingly unlikely) assumption
> that
> >>> 200
> >>>>     >> busy threads mean you have 200 threads processing long
running
> >>>>     requests.
> >>>>     >> Had you provided the connector configuration I asked for
(by
> that
> >>>>     I mean
> >>>>     >> the <Connector ... /> elements in server.xml) then
I'd be in a
> >>> better
> >>>>     >> position to tell you what is wrong.
> >>>>     >>
> >>>>     >> If you do have 200 long running requests then the
> >>>>     >> StuckThreadDetectionValve is how you detect them. That
fact
> that
> >>> that
> >>>>     >> valve is not reporting any long running requests should
be a
> big
> >>> clue
> >>>>     >> that your assumptions about what is going on are not correct.
> >>>>     >>
> >>>>     >> If, and it is a big if, you did have 200 concurrent long
> running
> >>>>     >> requests then there is no guaranteed way to stop them.
If your
> >>>>     >> application checks for interrupts (most don't) then the
> >>>>     >> StuckThreadDetectionValve can interrupt them which should
> >>>>     terminate the
> >>>>     >> processing of that request but the time it would take to
code
> the
> >>>>     >> application to handle that properly would normally be better
> >>> spent
> >>>>     >> fixing the root cause of the long running request.
> >>>>     >>
> >>>>     >> Mark
> >>>>     >>
> >>>>     >>>
> >>>>     >>>
> >>>>     >>>
> >>>>     >>>
> >>>>     >>>
> >>>>     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
> >>>>     <mailto:markt@apache.org>> wrote:
> >>>>     >>>
> >>>>     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
> >>>>     >>>>>         We are facing issues with long running
thread in
> >>>>     tomcat . we
> >>>>     >> are
> >>>>     >>>>> using Tomcat-7.0.47.Tomcat manager shows Current
busy
> threads
> >>>>     : 200,
> >>>>     >>>>> application  gets stucked due to these long
running threads.
> >>>>     >>>>
> >>>>     >>>> What makes you think you have issues with long
running
> threads?
> >>>>     A busy
> >>>>     >>>> thread isn't necessarily processing a request.
Configuration
> >>> errors
> >>>>     >>>> leading to thread starvation are a more typical
cause of the
> >>>>     symptom you
> >>>>     >>>> describe rather than long running threads in the
application.
> >>>>     >>>>
> >>>>     >>>>>          We implemented StuckThreadDetectionValve
in
> >>>>     server.xml( <Valve
> >>>>     >>>>>
> >>>>     >>>>>
> >>>>      className="org.apache.catalina.valves.StuckThreadDetectionValve"
> >>>>     >>>>>
> >>>>     >>>>>     threshold="60" />), but it  could not
help out.
> >>>>     >>>>
> >>>>     >>>> Why not? Again, this suggests that long running
threads
> aren't
> >>> the
> >>>>     >> problem.
> >>>>     >>>>
> >>>>     >>>>>        So we implemented custom StuckThreadDetectionValve
by
> >>>>     extending
> >>>>     >>>>> StuckThreadDetectionValve from
> >>>>     >>>>> catalina, it only goes to "constructor","initInternal",and
> in
> >>>>     >>>> "invoke"(when
> >>>>     >>>>> action fires), it does not go to function
> >>>>     "getStuckThreadNames()" even
> >>>>     >>>>> after threshold time.How to achieve the same.Is
there any
> way
> >>>>     to detect
> >>>>     >>>>> stucked thread and kill them?
> >>>>     >>>>
> >>>>     >>>> First of all your invoke() method fails to call
> super.invoke()
> >>>>     so the
> >>>>     >>>> Valve is never going to do anything.
> >>>>     >>>>
> >>>>     >>>> Secondly, if the original valve didn't work adding
a bunch of
> >>>>     >>>> System.out.println() lines isn't going to magically
make it
> >>> work.
> >>>>     >>>>
> >>>>     >>>> Thirdly, getStuckThreadNames() is never called
by any Tomcat
> >>>>     code so it
> >>>>     >>>> is no surprise that you never see a call to that
method. That
> >>>>     method is
> >>>>     >>>> intended for use via JMX.
> >>>>     >>>>
> >>>>     >>>> I suggest you start again and tell us more about
the problem
> >>>>     you are
> >>>>     >>>> trying to solve (lack of response) and your configuration.
In
> >>>>     particular
> >>>>     >>>> we need to know your connector configuration and
how users
> >>>>     access the
> >>>>     >>>> application. Do they connect directly to Tomcat
or do they go
> >>> via a
> >>>>     >>>> reverse proxy?
> >>>>     >>>>
> >>>>     >>>> Mark
> >>>>     >>>>
> >>>>     >>>>
> >>>>     >>>>>
> >>>>     >>>>>   My custom Valve is like following :
> >>>>     >>>>>
> >>>>     >>>>> public class StuckThreadDetection extends
> >>>>     StuckThreadDetectionValve
> >>>>     >>>>> {
> >>>>     >>>>> StuckThreadDetection stuckThreadDetection;
> >>>>     >>>>>
> >>>>     >>>>> public StuckThreadDetection()
> >>>>     >>>>> {
> >>>>     >>>>> System.out.println("in StuckThreadDetection
constructor");
> >>>>     >>>>>
> >>>>     >>>>> }
> >>>>     >>>>>
> >>>>     >>>>> public void invoke(Request request, Response
response)
> throws
> >>>>     >>>> IOException,
> >>>>     >>>>> ServletException
> >>>>     >>>>> {
> >>>>     >>>>> System.out.println("in invoke...");
> >>>>     >>>>>
> >>>>     >>>>> getNext().invoke(request, response);
> >>>>     >>>>> }
> >>>>     >>>>>
> >>>>     >>>>> @Override
> >>>>     >>>>> protected void initInternal() throws LifecycleException
> >>>>     >>>>> {
> >>>>     >>>>> System.out.println("In initInternal");
> >>>>     >>>>> super.initInternal();
> >>>>     >>>>> }
> >>>>     >>>>>
> >>>>     >>>>> @Override
> >>>>     >>>>> public String[] getStuckThreadNames()
> >>>>     >>>>> {
> >>>>     >>>>> System.out.println("in getStuckThreadNames...");
> >>>>     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
> >>>>     >>>>>
> >>>>     >>>>> for (String threadName : listStuckedThread)
> >>>>     >>>>> {
> >>>>     >>>>> System.out.println(threadName);
> >>>>     >>>>> }
> >>>>     >>>>>
> >>>>     >>>>> return listStuckedThread;
> >>>>     >>>>>
> >>>>     >>>>> }
> >>>>     >>>>>
> >>>>     >>>>> @Override
> >>>>     >>>>> public String getInfo()
> >>>>     >>>>> {
> >>>>     >>>>> System.out.println("In getInfo");
> >>>>     >>>>> return super.getInfo();
> >>>>     >>>>> }
> >>>>     >>>>> }
> >>>>     >>>>>
> >>>>     >>>>>
> >>>>     >>>>>
> >>>>     >>>>>
> >>>>     >>>>
> >>>>     >>>>
> >>>>     >>>>
> >>>>
> >>>  ---------------------------------------------------------------------
> >>>>     >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>     >>>> For additional commands, e-mail:
> users-help@tomcat.apache.org
> >>>>     <mailto:users-help@tomcat.apache.org>
> >>>>     >>>>
> >>>>     >>>>
> >>>>     >>>
> >>>>     >>>
> >>>>     >>
> >>>>     >>
> >>>>     >>
> >>> ---------------------------------------------------------------------
> >>>>     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>     >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>     <mailto:users-help@tomcat.apache.org>
> >>>>     >>
> >>>>     >>
> >>>>     >
> >>>>     >
> >>>>
> >>>>
> >>>>
> >>>  ---------------------------------------------------------------------
> >>>>     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>     For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>     <mailto:users-help@tomcat.apache.org>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> /Thanks & Regards,/
> >>>> / /
> >>>> / Yogesh Patel/
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>
> >>>
> >>
> >>
> >> --
> >> *Thanks & Regards,*
> >>
> >> * Yogesh Patel*
> >>
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


-- 
*Thanks & Regards,*

* Yogesh Patel*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message