tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: Regarding StuckThreadDetectionValve
Date Mon, 05 Oct 2015 19:37:04 GMT
On 05/10/2015 16:00, Yogesh Patel wrote:
> Other 199 process are also for /solr430/update?wt=xml&version=2.2 HTTP/1.1
> only

That is surprising. That could be a Solr issue or a Tomcat issue. Best
to take three thread dumps each 10 to 30 seconds apart and then take a
look to see what is holding up those threads. Best guess is a deadlock
or a database issue.

(You still have the potential for thread starvation that will come back
to bite you later after you fix this issue unless you fix it)

Mark

> 
> On 5 October 2015 at 20:18, Mark Thomas <markt@apache.org> wrote:
> 
>> On 05/10/2015 12:58, Yogesh Patel wrote:
>>> Hi Mark Thomas,
>>>
>>> in image it shows Tomcat Manager screen:
>>> Under "ajp-apr-10003" Section in tomcat manager:
>>> Max threads: 200 Current thread count:200 Current thread busy :200 Keeped
>>> alive socket count :0
>>>
>>> *Stage   Time              BSent    BRecv          Client         VHost
>>>  Request*
>>> S          2884466 ms    0 KB     184063 KB     machineip     host
>>  POST
>>> /solr430/update?wt=xml&version=2.2 HTTP/1.1
>>
>> And the other 199 processors?
>>
>> Mark
>>
>>
>>>
>>> ..........
>>> ............
>>>
>>>
>>> On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.patel@highq.com>
>> wrote:
>>>
>>>> Hi , let me try sending image using attachment , if still its not
>> viewable
>>>> then i will find another way.
>>>>
>>>> On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:
>>>>
>>>>> On 05/10/2015 11:28, Yogesh Patel wrote:
>>>>>> Hi Thomas ,
>>>>>>                   Please see this image ...have a look at Time column
>>>>>
>>>>> The list strips images. If you really want us to look at the image (not
>>>>> that I think it will be remotely relevant) upload it somewhere and post
>>>>> the URL (make sure it is publicly accessible and doe snot require any
>>>>> form of registration to view it).
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> ‚Äč
>>>>>>
>>>>>> On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
>>>>>> <mailto:markt@apache.org>> wrote:
>>>>>>
>>>>>>     On 05/10/2015 10:09, Yogesh Patel wrote:
>>>>>>     > Hi Thomas,
>>>>>>     >
>>>>>>     > Connector configuration is like :
>>>>>>     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443"
>> />
>>>>>>
>>>>>>     That means you are using the BIO AJP connector.
>>>>>>
>>>>>>     You don't have a problem with long running requests.
>>>>>>
>>>>>>     You have a problem with thread starvation.
>>>>>>
>>>>>>     AJP uses persistent connections.
>>>>>>
>>>>>>     BIO requires one thread per connection, regardless of whether
or
>> nor
>>>>>>     that connection is processing a request or waiting for the next
>> one.
>>>>>>
>>>>>>     httpd will (eventually, assuming mostly default config) create
one
>>>>> AJP
>>>>>>     connection for each httpd thread. If you have more httpd threads
>>>>> than
>>>>>>     Tomcat threads you will eventually reach the point where httpd
>> can't
>>>>>>     create a Tomcat thread it requires and at that point it will
>> appear
>>>>>>     to hang.
>>>>>>
>>>>>>     There are several possible fixes:
>>>>>>     a) disable connection re-use in httpd for AJP connections
>>>>>>     b) use the NIO AJP connector in Tomcat
>>>>>>     c) increase maxThreads in Tomcat so it is > max threads in
httpd
>>>>>>
>>>>>>     For more explanation read this:
>>>>>>
>>>>>
>> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>>>>>>     <
>>>>>
>> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>>>>>>
>>>>>>
>>>>>>     from slide 29 to 33
>>>>>>
>>>>>>     Mark
>>>>>>
>>>>>>
>>>>>>
>>>>>>     >
>>>>>>     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
>>>>>>     <mailto:markt@apache.org>> wrote:
>>>>>>     >
>>>>>>     >> On 05/10/2015 09:07, Yogesh Patel wrote:
>>>>>>     >>> Thanks Mark Thomas ,
>>>>>>     >>>
>>>>>>     >>>    Our application is access by Apache TO Tomcat
using AJP
>>>>>>     Connector.
>>>>>>     >>
>>>>>>     >> OK. That answers one of the questions I asked. However,
you
>>>>> haven't
>>>>>>     >> provided the connector configuration.
>>>>>>     >>
>>>>>>     >>>     Problem :
>>>>>>     >>>     Our application was mostly hanged,after looking
at tomcat
>>>>>>     manager it
>>>>>>     >>> shown there are so many long running threads shown.
>>>>>>     >>
>>>>>>     >> The Tomcat Manager app does not show long running threads.
It
>>>>>>     does show
>>>>>>     >> you how many requests are busy but again, a busy thread
is not
>>>>>>     >> necessarily processing a request.
>>>>>>     >>
>>>>>>     >>>     We want to recognize why so many threads are
running since
>>>>>>     long time.
>>>>>>     >>
>>>>>>     >> Threads are always "running". The key question is are
those
>>>>> threads
>>>>>>     >> 'idle' or 'busy'. An idle thread is waiting to be allocated
>> work.
>>>>>>     A busy
>>>>>>     >> thread has been allocated work but the manager application
>> can't
>>>>>>     >> distinguish if that work is 'wait for a request to process'
or
>>>>>>     >> 'processing a request'.
>>>>>>     >>
>>>>>>     >>>     We want to detect such thread and want to kill
these
>> stucked
>>>>>>     thread.
>>>>>>     >>
>>>>>>     >> You continue to make the (increasingly unlikely) assumption
>> that
>>>>> 200
>>>>>>     >> busy threads mean you have 200 threads processing long
running
>>>>>>     requests.
>>>>>>     >> Had you provided the connector configuration I asked
for (by
>> that
>>>>>>     I mean
>>>>>>     >> the <Connector ... /> elements in server.xml)
then I'd be in a
>>>>> better
>>>>>>     >> position to tell you what is wrong.
>>>>>>     >>
>>>>>>     >> If you do have 200 long running requests then the
>>>>>>     >> StuckThreadDetectionValve is how you detect them. That
fact
>> that
>>>>> that
>>>>>>     >> valve is not reporting any long running requests should
be a
>> big
>>>>> clue
>>>>>>     >> that your assumptions about what is going on are not
correct.
>>>>>>     >>
>>>>>>     >> If, and it is a big if, you did have 200 concurrent
long
>> running
>>>>>>     >> requests then there is no guaranteed way to stop them.
If your
>>>>>>     >> application checks for interrupts (most don't) then
the
>>>>>>     >> StuckThreadDetectionValve can interrupt them which should
>>>>>>     terminate the
>>>>>>     >> processing of that request but the time it would take
to code
>> the
>>>>>>     >> application to handle that properly would normally be
better
>>>>> spent
>>>>>>     >> fixing the root cause of the long running request.
>>>>>>     >>
>>>>>>     >> Mark
>>>>>>     >>
>>>>>>     >>>
>>>>>>     >>>
>>>>>>     >>>
>>>>>>     >>>
>>>>>>     >>>
>>>>>>     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
>>>>>>     <mailto:markt@apache.org>> wrote:
>>>>>>     >>>
>>>>>>     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
>>>>>>     >>>>>         We are facing issues with long running
thread in
>>>>>>     tomcat . we
>>>>>>     >> are
>>>>>>     >>>>> using Tomcat-7.0.47.Tomcat manager shows
Current busy
>> threads
>>>>>>     : 200,
>>>>>>     >>>>> application  gets stucked due to these long
running threads.
>>>>>>     >>>>
>>>>>>     >>>> What makes you think you have issues with long
running
>> threads?
>>>>>>     A busy
>>>>>>     >>>> thread isn't necessarily processing a request.
Configuration
>>>>> errors
>>>>>>     >>>> leading to thread starvation are a more typical
cause of the
>>>>>>     symptom you
>>>>>>     >>>> describe rather than long running threads in
the application.
>>>>>>     >>>>
>>>>>>     >>>>>          We implemented StuckThreadDetectionValve
in
>>>>>>     server.xml( <Valve
>>>>>>     >>>>>
>>>>>>     >>>>>
>>>>>>      className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>>>     >>>>>
>>>>>>     >>>>>     threshold="60" />), but it  could
not help out.
>>>>>>     >>>>
>>>>>>     >>>> Why not? Again, this suggests that long running
threads
>> aren't
>>>>> the
>>>>>>     >> problem.
>>>>>>     >>>>
>>>>>>     >>>>>        So we implemented custom StuckThreadDetectionValve
by
>>>>>>     extending
>>>>>>     >>>>> StuckThreadDetectionValve from
>>>>>>     >>>>> catalina, it only goes to "constructor","initInternal",and
>> in
>>>>>>     >>>> "invoke"(when
>>>>>>     >>>>> action fires), it does not go to function
>>>>>>     "getStuckThreadNames()" even
>>>>>>     >>>>> after threshold time.How to achieve the
same.Is there any
>> way
>>>>>>     to detect
>>>>>>     >>>>> stucked thread and kill them?
>>>>>>     >>>>
>>>>>>     >>>> First of all your invoke() method fails to call
>> super.invoke()
>>>>>>     so the
>>>>>>     >>>> Valve is never going to do anything.
>>>>>>     >>>>
>>>>>>     >>>> Secondly, if the original valve didn't work
adding a bunch of
>>>>>>     >>>> System.out.println() lines isn't going to magically
make it
>>>>> work.
>>>>>>     >>>>
>>>>>>     >>>> Thirdly, getStuckThreadNames() is never called
by any Tomcat
>>>>>>     code so it
>>>>>>     >>>> is no surprise that you never see a call to
that method. That
>>>>>>     method is
>>>>>>     >>>> intended for use via JMX.
>>>>>>     >>>>
>>>>>>     >>>> I suggest you start again and tell us more about
the problem
>>>>>>     you are
>>>>>>     >>>> trying to solve (lack of response) and your
configuration. In
>>>>>>     particular
>>>>>>     >>>> we need to know your connector configuration
and how users
>>>>>>     access the
>>>>>>     >>>> application. Do they connect directly to Tomcat
or do they go
>>>>> via a
>>>>>>     >>>> reverse proxy?
>>>>>>     >>>>
>>>>>>     >>>> Mark
>>>>>>     >>>>
>>>>>>     >>>>
>>>>>>     >>>>>
>>>>>>     >>>>>   My custom Valve is like following :
>>>>>>     >>>>>
>>>>>>     >>>>> public class StuckThreadDetection extends
>>>>>>     StuckThreadDetectionValve
>>>>>>     >>>>> {
>>>>>>     >>>>> StuckThreadDetection stuckThreadDetection;
>>>>>>     >>>>>
>>>>>>     >>>>> public StuckThreadDetection()
>>>>>>     >>>>> {
>>>>>>     >>>>> System.out.println("in StuckThreadDetection
constructor");
>>>>>>     >>>>>
>>>>>>     >>>>> }
>>>>>>     >>>>>
>>>>>>     >>>>> public void invoke(Request request, Response
response)
>> throws
>>>>>>     >>>> IOException,
>>>>>>     >>>>> ServletException
>>>>>>     >>>>> {
>>>>>>     >>>>> System.out.println("in invoke...");
>>>>>>     >>>>>
>>>>>>     >>>>> getNext().invoke(request, response);
>>>>>>     >>>>> }
>>>>>>     >>>>>
>>>>>>     >>>>> @Override
>>>>>>     >>>>> protected void initInternal() throws LifecycleException
>>>>>>     >>>>> {
>>>>>>     >>>>> System.out.println("In initInternal");
>>>>>>     >>>>> super.initInternal();
>>>>>>     >>>>> }
>>>>>>     >>>>>
>>>>>>     >>>>> @Override
>>>>>>     >>>>> public String[] getStuckThreadNames()
>>>>>>     >>>>> {
>>>>>>     >>>>> System.out.println("in getStuckThreadNames...");
>>>>>>     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
>>>>>>     >>>>>
>>>>>>     >>>>> for (String threadName : listStuckedThread)
>>>>>>     >>>>> {
>>>>>>     >>>>> System.out.println(threadName);
>>>>>>     >>>>> }
>>>>>>     >>>>>
>>>>>>     >>>>> return listStuckedThread;
>>>>>>     >>>>>
>>>>>>     >>>>> }
>>>>>>     >>>>>
>>>>>>     >>>>> @Override
>>>>>>     >>>>> public String getInfo()
>>>>>>     >>>>> {
>>>>>>     >>>>> System.out.println("In getInfo");
>>>>>>     >>>>> return super.getInfo();
>>>>>>     >>>>> }
>>>>>>     >>>>> }
>>>>>>     >>>>>
>>>>>>     >>>>>
>>>>>>     >>>>>
>>>>>>     >>>>>
>>>>>>     >>>>
>>>>>>     >>>>
>>>>>>     >>>>
>>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>>     >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
>>>>>>     >>>> For additional commands, e-mail:
>> users-help@tomcat.apache.org
>>>>>>     <mailto:users-help@tomcat.apache.org>
>>>>>>     >>>>
>>>>>>     >>>>
>>>>>>     >>>
>>>>>>     >>>
>>>>>>     >>
>>>>>>     >>
>>>>>>     >>
>>>>> ---------------------------------------------------------------------
>>>>>>     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
>>>>>>     >> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>     <mailto:users-help@tomcat.apache.org>
>>>>>>     >>
>>>>>>     >>
>>>>>>     >
>>>>>>     >
>>>>>>
>>>>>>
>>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>>     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
>>>>>>     For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>     <mailto:users-help@tomcat.apache.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> /Thanks & Regards,/
>>>>>> / /
>>>>>> / Yogesh Patel/
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards,*
>>>>
>>>> * Yogesh Patel*
>>>>
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message