tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogesh Patel <yogesh.r.pa...@highq.com>
Subject Re: Regarding StuckThreadDetectionValve
Date Tue, 06 Oct 2015 06:06:27 GMT
Thanks Thomas,

            I just wanted to know in tomcat is there any way to detect such
long running thread and kill them.

On 6 October 2015 at 01:07, Mark Thomas <markt@apache.org> wrote:

> On 05/10/2015 16:00, Yogesh Patel wrote:
> > Other 199 process are also for /solr430/update?wt=xml&version=2.2
> HTTP/1.1
> > only
>
> That is surprising. That could be a Solr issue or a Tomcat issue. Best
> to take three thread dumps each 10 to 30 seconds apart and then take a
> look to see what is holding up those threads. Best guess is a deadlock
> or a database issue.
>
> (You still have the potential for thread starvation that will come back
> to bite you later after you fix this issue unless you fix it)
>
> Mark
>
> >
> > On 5 October 2015 at 20:18, Mark Thomas <markt@apache.org> wrote:
> >
> >> On 05/10/2015 12:58, Yogesh Patel wrote:
> >>> Hi Mark Thomas,
> >>>
> >>> in image it shows Tomcat Manager screen:
> >>> Under "ajp-apr-10003" Section in tomcat manager:
> >>> Max threads: 200 Current thread count:200 Current thread busy :200
> Keeped
> >>> alive socket count :0
> >>>
> >>> *Stage   Time              BSent    BRecv          Client         VHost
> >>>  Request*
> >>> S          2884466 ms    0 KB     184063 KB     machineip     host
> >>  POST
> >>> /solr430/update?wt=xml&version=2.2 HTTP/1.1
> >>
> >> And the other 199 processors?
> >>
> >> Mark
> >>
> >>
> >>>
> >>> ..........
> >>> ............
> >>>
> >>>
> >>> On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.patel@highq.com>
> >> wrote:
> >>>
> >>>> Hi , let me try sending image using attachment , if still its not
> >> viewable
> >>>> then i will find another way.
> >>>>
> >>>> On 5 October 2015 at 16:06, Mark Thomas <markt@apache.org> wrote:
> >>>>
> >>>>> On 05/10/2015 11:28, Yogesh Patel wrote:
> >>>>>> Hi Thomas ,
> >>>>>>                   Please see this image ...have a look at Time
> column
> >>>>>
> >>>>> The list strips images. If you really want us to look at the image
> (not
> >>>>> that I think it will be remotely relevant) upload it somewhere and
> post
> >>>>> the URL (make sure it is publicly accessible and doe snot require
any
> >>>>> form of registration to view it).
> >>>>>
> >>>>> Mark
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> ‚Äč
> >>>>>>
> >>>>>> On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
> >>>>>> <mailto:markt@apache.org>> wrote:
> >>>>>>
> >>>>>>     On 05/10/2015 10:09, Yogesh Patel wrote:
> >>>>>>     > Hi Thomas,
> >>>>>>     >
> >>>>>>     > Connector configuration is like :
> >>>>>>     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443"
> >> />
> >>>>>>
> >>>>>>     That means you are using the BIO AJP connector.
> >>>>>>
> >>>>>>     You don't have a problem with long running requests.
> >>>>>>
> >>>>>>     You have a problem with thread starvation.
> >>>>>>
> >>>>>>     AJP uses persistent connections.
> >>>>>>
> >>>>>>     BIO requires one thread per connection, regardless of whether
or
> >> nor
> >>>>>>     that connection is processing a request or waiting for the
next
> >> one.
> >>>>>>
> >>>>>>     httpd will (eventually, assuming mostly default config)
create
> one
> >>>>> AJP
> >>>>>>     connection for each httpd thread. If you have more httpd
threads
> >>>>> than
> >>>>>>     Tomcat threads you will eventually reach the point where
httpd
> >> can't
> >>>>>>     create a Tomcat thread it requires and at that point it
will
> >> appear
> >>>>>>     to hang.
> >>>>>>
> >>>>>>     There are several possible fixes:
> >>>>>>     a) disable connection re-use in httpd for AJP connections
> >>>>>>     b) use the NIO AJP connector in Tomcat
> >>>>>>     c) increase maxThreads in Tomcat so it is > max threads
in httpd
> >>>>>>
> >>>>>>     For more explanation read this:
> >>>>>>
> >>>>>
> >>
> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >>>>>>     <
> >>>>>
> >>
> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >>>>>>
> >>>>>>
> >>>>>>     from slide 29 to 33
> >>>>>>
> >>>>>>     Mark
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>     >
> >>>>>>     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
> >>>>>>     <mailto:markt@apache.org>> wrote:
> >>>>>>     >
> >>>>>>     >> On 05/10/2015 09:07, Yogesh Patel wrote:
> >>>>>>     >>> Thanks Mark Thomas ,
> >>>>>>     >>>
> >>>>>>     >>>    Our application is access by Apache TO Tomcat
using AJP
> >>>>>>     Connector.
> >>>>>>     >>
> >>>>>>     >> OK. That answers one of the questions I asked.
However, you
> >>>>> haven't
> >>>>>>     >> provided the connector configuration.
> >>>>>>     >>
> >>>>>>     >>>     Problem :
> >>>>>>     >>>     Our application was mostly hanged,after
looking at
> tomcat
> >>>>>>     manager it
> >>>>>>     >>> shown there are so many long running threads
shown.
> >>>>>>     >>
> >>>>>>     >> The Tomcat Manager app does not show long running
threads. It
> >>>>>>     does show
> >>>>>>     >> you how many requests are busy but again, a busy
thread is
> not
> >>>>>>     >> necessarily processing a request.
> >>>>>>     >>
> >>>>>>     >>>     We want to recognize why so many threads
are running
> since
> >>>>>>     long time.
> >>>>>>     >>
> >>>>>>     >> Threads are always "running". The key question
is are those
> >>>>> threads
> >>>>>>     >> 'idle' or 'busy'. An idle thread is waiting to
be allocated
> >> work.
> >>>>>>     A busy
> >>>>>>     >> thread has been allocated work but the manager
application
> >> can't
> >>>>>>     >> distinguish if that work is 'wait for a request
to process'
> or
> >>>>>>     >> 'processing a request'.
> >>>>>>     >>
> >>>>>>     >>>     We want to detect such thread and want
to kill these
> >> stucked
> >>>>>>     thread.
> >>>>>>     >>
> >>>>>>     >> You continue to make the (increasingly unlikely)
assumption
> >> that
> >>>>> 200
> >>>>>>     >> busy threads mean you have 200 threads processing
long
> running
> >>>>>>     requests.
> >>>>>>     >> Had you provided the connector configuration I
asked for (by
> >> that
> >>>>>>     I mean
> >>>>>>     >> the <Connector ... /> elements in server.xml)
then I'd be in
> a
> >>>>> better
> >>>>>>     >> position to tell you what is wrong.
> >>>>>>     >>
> >>>>>>     >> If you do have 200 long running requests then the
> >>>>>>     >> StuckThreadDetectionValve is how you detect them.
That fact
> >> that
> >>>>> that
> >>>>>>     >> valve is not reporting any long running requests
should be a
> >> big
> >>>>> clue
> >>>>>>     >> that your assumptions about what is going on are
not correct.
> >>>>>>     >>
> >>>>>>     >> If, and it is a big if, you did have 200 concurrent
long
> >> running
> >>>>>>     >> requests then there is no guaranteed way to stop
them. If
> your
> >>>>>>     >> application checks for interrupts (most don't)
then the
> >>>>>>     >> StuckThreadDetectionValve can interrupt them which
should
> >>>>>>     terminate the
> >>>>>>     >> processing of that request but the time it would
take to code
> >> the
> >>>>>>     >> application to handle that properly would normally
be better
> >>>>> spent
> >>>>>>     >> fixing the root cause of the long running request.
> >>>>>>     >>
> >>>>>>     >> Mark
> >>>>>>     >>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
> >>>>>>     <mailto:markt@apache.org>> wrote:
> >>>>>>     >>>
> >>>>>>     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
> >>>>>>     >>>>>         We are facing issues with long
running thread in
> >>>>>>     tomcat . we
> >>>>>>     >> are
> >>>>>>     >>>>> using Tomcat-7.0.47.Tomcat manager
shows Current busy
> >> threads
> >>>>>>     : 200,
> >>>>>>     >>>>> application  gets stucked due to these
long running
> threads.
> >>>>>>     >>>>
> >>>>>>     >>>> What makes you think you have issues with
long running
> >> threads?
> >>>>>>     A busy
> >>>>>>     >>>> thread isn't necessarily processing a request.
> Configuration
> >>>>> errors
> >>>>>>     >>>> leading to thread starvation are a more
typical cause of
> the
> >>>>>>     symptom you
> >>>>>>     >>>> describe rather than long running threads
in the
> application.
> >>>>>>     >>>>
> >>>>>>     >>>>>          We implemented StuckThreadDetectionValve
in
> >>>>>>     server.xml( <Valve
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>
> className="org.apache.catalina.valves.StuckThreadDetectionValve"
> >>>>>>     >>>>>
> >>>>>>     >>>>>     threshold="60" />), but it 
could not help out.
> >>>>>>     >>>>
> >>>>>>     >>>> Why not? Again, this suggests that long
running threads
> >> aren't
> >>>>> the
> >>>>>>     >> problem.
> >>>>>>     >>>>
> >>>>>>     >>>>>        So we implemented custom StuckThreadDetectionValve
> by
> >>>>>>     extending
> >>>>>>     >>>>> StuckThreadDetectionValve from
> >>>>>>     >>>>> catalina, it only goes to "constructor","initInternal",and
> >> in
> >>>>>>     >>>> "invoke"(when
> >>>>>>     >>>>> action fires), it does not go to function
> >>>>>>     "getStuckThreadNames()" even
> >>>>>>     >>>>> after threshold time.How to achieve
the same.Is there any
> >> way
> >>>>>>     to detect
> >>>>>>     >>>>> stucked thread and kill them?
> >>>>>>     >>>>
> >>>>>>     >>>> First of all your invoke() method fails
to call
> >> super.invoke()
> >>>>>>     so the
> >>>>>>     >>>> Valve is never going to do anything.
> >>>>>>     >>>>
> >>>>>>     >>>> Secondly, if the original valve didn't
work adding a bunch
> of
> >>>>>>     >>>> System.out.println() lines isn't going
to magically make it
> >>>>> work.
> >>>>>>     >>>>
> >>>>>>     >>>> Thirdly, getStuckThreadNames() is never
called by any
> Tomcat
> >>>>>>     code so it
> >>>>>>     >>>> is no surprise that you never see a call
to that method.
> That
> >>>>>>     method is
> >>>>>>     >>>> intended for use via JMX.
> >>>>>>     >>>>
> >>>>>>     >>>> I suggest you start again and tell us more
about the
> problem
> >>>>>>     you are
> >>>>>>     >>>> trying to solve (lack of response) and
your configuration.
> In
> >>>>>>     particular
> >>>>>>     >>>> we need to know your connector configuration
and how users
> >>>>>>     access the
> >>>>>>     >>>> application. Do they connect directly to
Tomcat or do they
> go
> >>>>> via a
> >>>>>>     >>>> reverse proxy?
> >>>>>>     >>>>
> >>>>>>     >>>> Mark
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>>   My custom Valve is like following
:
> >>>>>>     >>>>>
> >>>>>>     >>>>> public class StuckThreadDetection extends
> >>>>>>     StuckThreadDetectionValve
> >>>>>>     >>>>> {
> >>>>>>     >>>>> StuckThreadDetection stuckThreadDetection;
> >>>>>>     >>>>>
> >>>>>>     >>>>> public StuckThreadDetection()
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("in StuckThreadDetection
constructor");
> >>>>>>     >>>>>
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> public void invoke(Request request,
Response response)
> >> throws
> >>>>>>     >>>> IOException,
> >>>>>>     >>>>> ServletException
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("in invoke...");
> >>>>>>     >>>>>
> >>>>>>     >>>>> getNext().invoke(request, response);
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> @Override
> >>>>>>     >>>>> protected void initInternal() throws
LifecycleException
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("In initInternal");
> >>>>>>     >>>>> super.initInternal();
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> @Override
> >>>>>>     >>>>> public String[] getStuckThreadNames()
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("in getStuckThreadNames...");
> >>>>>>     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
> >>>>>>     >>>>>
> >>>>>>     >>>>> for (String threadName : listStuckedThread)
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println(threadName);
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> return listStuckedThread;
> >>>>>>     >>>>>
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>> @Override
> >>>>>>     >>>>> public String getInfo()
> >>>>>>     >>>>> {
> >>>>>>     >>>>> System.out.println("In getInfo");
> >>>>>>     >>>>> return super.getInfo();
> >>>>>>     >>>>> }
> >>>>>>     >>>>> }
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>>
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>>>     >>>> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> >>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>>>     >>>> For additional commands, e-mail:
> >> users-help@tomcat.apache.org
> >>>>>>     <mailto:users-help@tomcat.apache.org>
> >>>>>>     >>>>
> >>>>>>     >>>>
> >>>>>>     >>>
> >>>>>>     >>>
> >>>>>>     >>
> >>>>>>     >>
> >>>>>>     >>
> >>>>> ---------------------------------------------------------------------
> >>>>>>     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>>>     >> For additional commands, e-mail:
> users-help@tomcat.apache.org
> >>>>>>     <mailto:users-help@tomcat.apache.org>
> >>>>>>     >>
> >>>>>>     >>
> >>>>>>     >
> >>>>>>     >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>>>     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>     <mailto:users-unsubscribe@tomcat.apache.org>
> >>>>>>     For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>     <mailto:users-help@tomcat.apache.org>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> /Thanks & Regards,/
> >>>>>> / /
> >>>>>> / Yogesh Patel/
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Thanks & Regards,*
> >>>>
> >>>> * Yogesh Patel*
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


-- 
*Thanks & Regards,*

* Yogesh Patel*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message