tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: Regarding StuckThreadDetectionValve
Date Mon, 05 Oct 2015 14:48:26 GMT
On 05/10/2015 12:58, Yogesh Patel wrote:
> Hi Mark Thomas,
> 
> in image it shows Tomcat Manager screen:
> Under "ajp-apr-10003" Section in tomcat manager:
> Max threads: 200 Current thread count:200 Current thread busy :200 Keeped
> alive socket count :0
> 
> *Stage   Time              BSent    BRecv          Client         VHost
>  Request*
> S          2884466 ms    0 KB     184063 KB     machineip     host     POST
> /solr430/update?wt=xml&version=2.2 HTTP/1.1

And the other 199 processors?

Mark


> 
> ..........
> ............
> 
> 
> On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.patel@highq.com> wrote:
> 
>> Hi , let me try sending image using attachment , if still its not viewable
>> then i will find another way.
>>
>> On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:
>>
>>> On 05/10/2015 11:28, Yogesh Patel wrote:
>>>> Hi Thomas ,
>>>>                   Please see this image ...have a look at Time column
>>>
>>> The list strips images. If you really want us to look at the image (not
>>> that I think it will be remotely relevant) upload it somewhere and post
>>> the URL (make sure it is publicly accessible and doe snot require any
>>> form of registration to view it).
>>>
>>> Mark
>>>
>>>
>>>>
>>>>
>>>> ‚Äč
>>>>
>>>> On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
>>>> <mailto:markt@apache.org>> wrote:
>>>>
>>>>     On 05/10/2015 10:09, Yogesh Patel wrote:
>>>>     > Hi Thomas,
>>>>     >
>>>>     > Connector configuration is like :
>>>>     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443"
/>
>>>>
>>>>     That means you are using the BIO AJP connector.
>>>>
>>>>     You don't have a problem with long running requests.
>>>>
>>>>     You have a problem with thread starvation.
>>>>
>>>>     AJP uses persistent connections.
>>>>
>>>>     BIO requires one thread per connection, regardless of whether or nor
>>>>     that connection is processing a request or waiting for the next one.
>>>>
>>>>     httpd will (eventually, assuming mostly default config) create one
>>> AJP
>>>>     connection for each httpd thread. If you have more httpd threads
>>> than
>>>>     Tomcat threads you will eventually reach the point where httpd can't
>>>>     create a Tomcat thread it requires and at that point it will appear
>>>>     to hang.
>>>>
>>>>     There are several possible fixes:
>>>>     a) disable connection re-use in httpd for AJP connections
>>>>     b) use the NIO AJP connector in Tomcat
>>>>     c) increase maxThreads in Tomcat so it is > max threads in httpd
>>>>
>>>>     For more explanation read this:
>>>>
>>> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>>>>     <
>>> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>>>>
>>>>
>>>>     from slide 29 to 33
>>>>
>>>>     Mark
>>>>
>>>>
>>>>
>>>>     >
>>>>     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
>>>>     <mailto:markt@apache.org>> wrote:
>>>>     >
>>>>     >> On 05/10/2015 09:07, Yogesh Patel wrote:
>>>>     >>> Thanks Mark Thomas ,
>>>>     >>>
>>>>     >>>    Our application is access by Apache TO Tomcat using AJP
>>>>     Connector.
>>>>     >>
>>>>     >> OK. That answers one of the questions I asked. However, you
>>> haven't
>>>>     >> provided the connector configuration.
>>>>     >>
>>>>     >>>     Problem :
>>>>     >>>     Our application was mostly hanged,after looking at tomcat
>>>>     manager it
>>>>     >>> shown there are so many long running threads shown.
>>>>     >>
>>>>     >> The Tomcat Manager app does not show long running threads. It
>>>>     does show
>>>>     >> you how many requests are busy but again, a busy thread is not
>>>>     >> necessarily processing a request.
>>>>     >>
>>>>     >>>     We want to recognize why so many threads are running
since
>>>>     long time.
>>>>     >>
>>>>     >> Threads are always "running". The key question is are those
>>> threads
>>>>     >> 'idle' or 'busy'. An idle thread is waiting to be allocated
work.
>>>>     A busy
>>>>     >> thread has been allocated work but the manager application can't
>>>>     >> distinguish if that work is 'wait for a request to process'
or
>>>>     >> 'processing a request'.
>>>>     >>
>>>>     >>>     We want to detect such thread and want to kill these
stucked
>>>>     thread.
>>>>     >>
>>>>     >> You continue to make the (increasingly unlikely) assumption
that
>>> 200
>>>>     >> busy threads mean you have 200 threads processing long running
>>>>     requests.
>>>>     >> Had you provided the connector configuration I asked for (by
that
>>>>     I mean
>>>>     >> the <Connector ... /> elements in server.xml) then I'd
be in a
>>> better
>>>>     >> position to tell you what is wrong.
>>>>     >>
>>>>     >> If you do have 200 long running requests then the
>>>>     >> StuckThreadDetectionValve is how you detect them. That fact
that
>>> that
>>>>     >> valve is not reporting any long running requests should be a
big
>>> clue
>>>>     >> that your assumptions about what is going on are not correct.
>>>>     >>
>>>>     >> If, and it is a big if, you did have 200 concurrent long running
>>>>     >> requests then there is no guaranteed way to stop them. If your
>>>>     >> application checks for interrupts (most don't) then the
>>>>     >> StuckThreadDetectionValve can interrupt them which should
>>>>     terminate the
>>>>     >> processing of that request but the time it would take to code
the
>>>>     >> application to handle that properly would normally be better
>>> spent
>>>>     >> fixing the root cause of the long running request.
>>>>     >>
>>>>     >> Mark
>>>>     >>
>>>>     >>>
>>>>     >>>
>>>>     >>>
>>>>     >>>
>>>>     >>>
>>>>     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
>>>>     <mailto:markt@apache.org>> wrote:
>>>>     >>>
>>>>     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
>>>>     >>>>>         We are facing issues with long running thread
in
>>>>     tomcat . we
>>>>     >> are
>>>>     >>>>> using Tomcat-7.0.47.Tomcat manager shows Current
busy threads
>>>>     : 200,
>>>>     >>>>> application  gets stucked due to these long running
threads.
>>>>     >>>>
>>>>     >>>> What makes you think you have issues with long running
threads?
>>>>     A busy
>>>>     >>>> thread isn't necessarily processing a request. Configuration
>>> errors
>>>>     >>>> leading to thread starvation are a more typical cause
of the
>>>>     symptom you
>>>>     >>>> describe rather than long running threads in the application.
>>>>     >>>>
>>>>     >>>>>          We implemented StuckThreadDetectionValve
in
>>>>     server.xml( <Valve
>>>>     >>>>>
>>>>     >>>>>
>>>>      className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>     >>>>>
>>>>     >>>>>     threshold="60" />), but it  could not help
out.
>>>>     >>>>
>>>>     >>>> Why not? Again, this suggests that long running threads
aren't
>>> the
>>>>     >> problem.
>>>>     >>>>
>>>>     >>>>>        So we implemented custom StuckThreadDetectionValve
by
>>>>     extending
>>>>     >>>>> StuckThreadDetectionValve from
>>>>     >>>>> catalina, it only goes to "constructor","initInternal",and
in
>>>>     >>>> "invoke"(when
>>>>     >>>>> action fires), it does not go to function
>>>>     "getStuckThreadNames()" even
>>>>     >>>>> after threshold time.How to achieve the same.Is
there any way
>>>>     to detect
>>>>     >>>>> stucked thread and kill them?
>>>>     >>>>
>>>>     >>>> First of all your invoke() method fails to call super.invoke()
>>>>     so the
>>>>     >>>> Valve is never going to do anything.
>>>>     >>>>
>>>>     >>>> Secondly, if the original valve didn't work adding a
bunch of
>>>>     >>>> System.out.println() lines isn't going to magically
make it
>>> work.
>>>>     >>>>
>>>>     >>>> Thirdly, getStuckThreadNames() is never called by any
Tomcat
>>>>     code so it
>>>>     >>>> is no surprise that you never see a call to that method.
That
>>>>     method is
>>>>     >>>> intended for use via JMX.
>>>>     >>>>
>>>>     >>>> I suggest you start again and tell us more about the
problem
>>>>     you are
>>>>     >>>> trying to solve (lack of response) and your configuration.
In
>>>>     particular
>>>>     >>>> we need to know your connector configuration and how
users
>>>>     access the
>>>>     >>>> application. Do they connect directly to Tomcat or do
they go
>>> via a
>>>>     >>>> reverse proxy?
>>>>     >>>>
>>>>     >>>> Mark
>>>>     >>>>
>>>>     >>>>
>>>>     >>>>>
>>>>     >>>>>   My custom Valve is like following :
>>>>     >>>>>
>>>>     >>>>> public class StuckThreadDetection extends
>>>>     StuckThreadDetectionValve
>>>>     >>>>> {
>>>>     >>>>> StuckThreadDetection stuckThreadDetection;
>>>>     >>>>>
>>>>     >>>>> public StuckThreadDetection()
>>>>     >>>>> {
>>>>     >>>>> System.out.println("in StuckThreadDetection constructor");
>>>>     >>>>>
>>>>     >>>>> }
>>>>     >>>>>
>>>>     >>>>> public void invoke(Request request, Response response)
throws
>>>>     >>>> IOException,
>>>>     >>>>> ServletException
>>>>     >>>>> {
>>>>     >>>>> System.out.println("in invoke...");
>>>>     >>>>>
>>>>     >>>>> getNext().invoke(request, response);
>>>>     >>>>> }
>>>>     >>>>>
>>>>     >>>>> @Override
>>>>     >>>>> protected void initInternal() throws LifecycleException
>>>>     >>>>> {
>>>>     >>>>> System.out.println("In initInternal");
>>>>     >>>>> super.initInternal();
>>>>     >>>>> }
>>>>     >>>>>
>>>>     >>>>> @Override
>>>>     >>>>> public String[] getStuckThreadNames()
>>>>     >>>>> {
>>>>     >>>>> System.out.println("in getStuckThreadNames...");
>>>>     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
>>>>     >>>>>
>>>>     >>>>> for (String threadName : listStuckedThread)
>>>>     >>>>> {
>>>>     >>>>> System.out.println(threadName);
>>>>     >>>>> }
>>>>     >>>>>
>>>>     >>>>> return listStuckedThread;
>>>>     >>>>>
>>>>     >>>>> }
>>>>     >>>>>
>>>>     >>>>> @Override
>>>>     >>>>> public String getInfo()
>>>>     >>>>> {
>>>>     >>>>> System.out.println("In getInfo");
>>>>     >>>>> return super.getInfo();
>>>>     >>>>> }
>>>>     >>>>> }
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>
>>>>     >>>>
>>>>     >>>>
>>>>
>>>  ---------------------------------------------------------------------
>>>>     >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
>>>>     >>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>     <mailto:users-help@tomcat.apache.org>
>>>>     >>>>
>>>>     >>>>
>>>>     >>>
>>>>     >>>
>>>>     >>
>>>>     >>
>>>>     >>
>>> ---------------------------------------------------------------------
>>>>     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
>>>>     >> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>     <mailto:users-help@tomcat.apache.org>
>>>>     >>
>>>>     >>
>>>>     >
>>>>     >
>>>>
>>>>
>>>>
>>>  ---------------------------------------------------------------------
>>>>     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
>>>>     For additional commands, e-mail: users-help@tomcat.apache.org
>>>>     <mailto:users-help@tomcat.apache.org>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> /Thanks & Regards,/
>>>> / /
>>>> / Yogesh Patel/
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards,*
>>
>> * Yogesh Patel*
>>
> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message