tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogesh Patel <yogesh.r.pa...@highq.com>
Subject Re: Regarding StuckThreadDetectionValve
Date Mon, 05 Oct 2015 11:58:42 GMT
Hi Mark Thomas,

in image it shows Tomcat Manager screen:
Under "ajp-apr-10003" Section in tomcat manager:
Max threads: 200 Current thread count:200 Current thread busy :200 Keeped
alive socket count :0

*Stage   Time              BSent    BRecv          Client         VHost
 Request*
S          2884466 ms    0 KB     184063 KB     machineip     host     POST
/solr430/update?wt=xml&version=2.2 HTTP/1.1

..........
............


On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.patel@highq.com> wrote:

> Hi , let me try sending image using attachment , if still its not viewable
> then i will find another way.
>
> On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:
>
>> On 05/10/2015 11:28, Yogesh Patel wrote:
>> > Hi Thomas ,
>> >                   Please see this image ...have a look at Time column
>>
>> The list strips images. If you really want us to look at the image (not
>> that I think it will be remotely relevant) upload it somewhere and post
>> the URL (make sure it is publicly accessible and doe snot require any
>> form of registration to view it).
>>
>> Mark
>>
>>
>> >
>> >
>> > ‚Äč
>> >
>> > On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
>> > <mailto:markt@apache.org>> wrote:
>> >
>> >     On 05/10/2015 10:09, Yogesh Patel wrote:
>> >     > Hi Thomas,
>> >     >
>> >     > Connector configuration is like :
>> >     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" />
>> >
>> >     That means you are using the BIO AJP connector.
>> >
>> >     You don't have a problem with long running requests.
>> >
>> >     You have a problem with thread starvation.
>> >
>> >     AJP uses persistent connections.
>> >
>> >     BIO requires one thread per connection, regardless of whether or nor
>> >     that connection is processing a request or waiting for the next one.
>> >
>> >     httpd will (eventually, assuming mostly default config) create one
>> AJP
>> >     connection for each httpd thread. If you have more httpd threads
>> than
>> >     Tomcat threads you will eventually reach the point where httpd can't
>> >     create a Tomcat thread it requires and at that point it will appear
>> >     to hang.
>> >
>> >     There are several possible fixes:
>> >     a) disable connection re-use in httpd for AJP connections
>> >     b) use the NIO AJP connector in Tomcat
>> >     c) increase maxThreads in Tomcat so it is > max threads in httpd
>> >
>> >     For more explanation read this:
>> >
>> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>> >     <
>> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>> >
>> >
>> >     from slide 29 to 33
>> >
>> >     Mark
>> >
>> >
>> >
>> >     >
>> >     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
>> >     <mailto:markt@apache.org>> wrote:
>> >     >
>> >     >> On 05/10/2015 09:07, Yogesh Patel wrote:
>> >     >>> Thanks Mark Thomas ,
>> >     >>>
>> >     >>>    Our application is access by Apache TO Tomcat using AJP
>> >     Connector.
>> >     >>
>> >     >> OK. That answers one of the questions I asked. However, you
>> haven't
>> >     >> provided the connector configuration.
>> >     >>
>> >     >>>     Problem :
>> >     >>>     Our application was mostly hanged,after looking at tomcat
>> >     manager it
>> >     >>> shown there are so many long running threads shown.
>> >     >>
>> >     >> The Tomcat Manager app does not show long running threads. It
>> >     does show
>> >     >> you how many requests are busy but again, a busy thread is not
>> >     >> necessarily processing a request.
>> >     >>
>> >     >>>     We want to recognize why so many threads are running since
>> >     long time.
>> >     >>
>> >     >> Threads are always "running". The key question is are those
>> threads
>> >     >> 'idle' or 'busy'. An idle thread is waiting to be allocated work.
>> >     A busy
>> >     >> thread has been allocated work but the manager application can't
>> >     >> distinguish if that work is 'wait for a request to process' or
>> >     >> 'processing a request'.
>> >     >>
>> >     >>>     We want to detect such thread and want to kill these stucked
>> >     thread.
>> >     >>
>> >     >> You continue to make the (increasingly unlikely) assumption that
>> 200
>> >     >> busy threads mean you have 200 threads processing long running
>> >     requests.
>> >     >> Had you provided the connector configuration I asked for (by that
>> >     I mean
>> >     >> the <Connector ... /> elements in server.xml) then I'd be
in a
>> better
>> >     >> position to tell you what is wrong.
>> >     >>
>> >     >> If you do have 200 long running requests then the
>> >     >> StuckThreadDetectionValve is how you detect them. That fact that
>> that
>> >     >> valve is not reporting any long running requests should be a big
>> clue
>> >     >> that your assumptions about what is going on are not correct.
>> >     >>
>> >     >> If, and it is a big if, you did have 200 concurrent long running
>> >     >> requests then there is no guaranteed way to stop them. If your
>> >     >> application checks for interrupts (most don't) then the
>> >     >> StuckThreadDetectionValve can interrupt them which should
>> >     terminate the
>> >     >> processing of that request but the time it would take to code the
>> >     >> application to handle that properly would normally be better
>> spent
>> >     >> fixing the root cause of the long running request.
>> >     >>
>> >     >> Mark
>> >     >>
>> >     >>>
>> >     >>>
>> >     >>>
>> >     >>>
>> >     >>>
>> >     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
>> >     <mailto:markt@apache.org>> wrote:
>> >     >>>
>> >     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
>> >     >>>>>         We are facing issues with long running thread
in
>> >     tomcat . we
>> >     >> are
>> >     >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy
threads
>> >     : 200,
>> >     >>>>> application  gets stucked due to these long running
threads.
>> >     >>>>
>> >     >>>> What makes you think you have issues with long running
threads?
>> >     A busy
>> >     >>>> thread isn't necessarily processing a request. Configuration
>> errors
>> >     >>>> leading to thread starvation are a more typical cause of
the
>> >     symptom you
>> >     >>>> describe rather than long running threads in the application.
>> >     >>>>
>> >     >>>>>          We implemented StuckThreadDetectionValve in
>> >     server.xml( <Valve
>> >     >>>>>
>> >     >>>>>
>> >      className="org.apache.catalina.valves.StuckThreadDetectionValve"
>> >     >>>>>
>> >     >>>>>     threshold="60" />), but it  could not help out.
>> >     >>>>
>> >     >>>> Why not? Again, this suggests that long running threads
aren't
>> the
>> >     >> problem.
>> >     >>>>
>> >     >>>>>        So we implemented custom StuckThreadDetectionValve
by
>> >     extending
>> >     >>>>> StuckThreadDetectionValve from
>> >     >>>>> catalina, it only goes to "constructor","initInternal",and
in
>> >     >>>> "invoke"(when
>> >     >>>>> action fires), it does not go to function
>> >     "getStuckThreadNames()" even
>> >     >>>>> after threshold time.How to achieve the same.Is there
any way
>> >     to detect
>> >     >>>>> stucked thread and kill them?
>> >     >>>>
>> >     >>>> First of all your invoke() method fails to call super.invoke()
>> >     so the
>> >     >>>> Valve is never going to do anything.
>> >     >>>>
>> >     >>>> Secondly, if the original valve didn't work adding a bunch
of
>> >     >>>> System.out.println() lines isn't going to magically make
it
>> work.
>> >     >>>>
>> >     >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat
>> >     code so it
>> >     >>>> is no surprise that you never see a call to that method.
That
>> >     method is
>> >     >>>> intended for use via JMX.
>> >     >>>>
>> >     >>>> I suggest you start again and tell us more about the problem
>> >     you are
>> >     >>>> trying to solve (lack of response) and your configuration.
In
>> >     particular
>> >     >>>> we need to know your connector configuration and how users
>> >     access the
>> >     >>>> application. Do they connect directly to Tomcat or do they
go
>> via a
>> >     >>>> reverse proxy?
>> >     >>>>
>> >     >>>> Mark
>> >     >>>>
>> >     >>>>
>> >     >>>>>
>> >     >>>>>   My custom Valve is like following :
>> >     >>>>>
>> >     >>>>> public class StuckThreadDetection extends
>> >     StuckThreadDetectionValve
>> >     >>>>> {
>> >     >>>>> StuckThreadDetection stuckThreadDetection;
>> >     >>>>>
>> >     >>>>> public StuckThreadDetection()
>> >     >>>>> {
>> >     >>>>> System.out.println("in StuckThreadDetection constructor");
>> >     >>>>>
>> >     >>>>> }
>> >     >>>>>
>> >     >>>>> public void invoke(Request request, Response response)
throws
>> >     >>>> IOException,
>> >     >>>>> ServletException
>> >     >>>>> {
>> >     >>>>> System.out.println("in invoke...");
>> >     >>>>>
>> >     >>>>> getNext().invoke(request, response);
>> >     >>>>> }
>> >     >>>>>
>> >     >>>>> @Override
>> >     >>>>> protected void initInternal() throws LifecycleException
>> >     >>>>> {
>> >     >>>>> System.out.println("In initInternal");
>> >     >>>>> super.initInternal();
>> >     >>>>> }
>> >     >>>>>
>> >     >>>>> @Override
>> >     >>>>> public String[] getStuckThreadNames()
>> >     >>>>> {
>> >     >>>>> System.out.println("in getStuckThreadNames...");
>> >     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
>> >     >>>>>
>> >     >>>>> for (String threadName : listStuckedThread)
>> >     >>>>> {
>> >     >>>>> System.out.println(threadName);
>> >     >>>>> }
>> >     >>>>>
>> >     >>>>> return listStuckedThread;
>> >     >>>>>
>> >     >>>>> }
>> >     >>>>>
>> >     >>>>> @Override
>> >     >>>>> public String getInfo()
>> >     >>>>> {
>> >     >>>>> System.out.println("In getInfo");
>> >     >>>>> return super.getInfo();
>> >     >>>>> }
>> >     >>>>> }
>> >     >>>>>
>> >     >>>>>
>> >     >>>>>
>> >     >>>>>
>> >     >>>>
>> >     >>>>
>> >     >>>>
>> >
>>  ---------------------------------------------------------------------
>> >     >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >     <mailto:users-unsubscribe@tomcat.apache.org>
>> >     >>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >     <mailto:users-help@tomcat.apache.org>
>> >     >>>>
>> >     >>>>
>> >     >>>
>> >     >>>
>> >     >>
>> >     >>
>> >     >>
>> ---------------------------------------------------------------------
>> >     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >     <mailto:users-unsubscribe@tomcat.apache.org>
>> >     >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >     <mailto:users-help@tomcat.apache.org>
>> >     >>
>> >     >>
>> >     >
>> >     >
>> >
>> >
>> >
>>  ---------------------------------------------------------------------
>> >     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >     <mailto:users-unsubscribe@tomcat.apache.org>
>> >     For additional commands, e-mail: users-help@tomcat.apache.org
>> >     <mailto:users-help@tomcat.apache.org>
>> >
>> >
>> >
>> >
>> > --
>> > /Thanks & Regards,/
>> > / /
>> > / Yogesh Patel/
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> --
> *Thanks & Regards,*
>
> * Yogesh Patel*
>



-- 
*Thanks & Regards,*

* Yogesh Patel*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message