Return-Path: 
 <users-return-254993-apmail-tomcat-users-archive=tomcat.apache.org@tomcat.apache.org>
X-Original-To: apmail-tomcat-users-archive@www.apache.org
Delivered-To: apmail-tomcat-users-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ED0261866C
	for <apmail-tomcat-users-archive@www.apache.org>;
 Mon,  5 Oct 2015 10:36:23 +0000 (UTC)
Received: (qmail 635 invoked by uid 500); 5 Oct 2015 10:36:21 -0000
Delivered-To: apmail-tomcat-users-archive@tomcat.apache.org
Received: (qmail 570 invoked by uid 500); 5 Oct 2015 10:36:21 -0000
Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@tomcat.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@tomcat.apache.org>
List-Post: <mailto:users@tomcat.apache.org>
List-Id: <users.tomcat.apache.org>
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Delivered-To: mailing list users@tomcat.apache.org
Received: (qmail 560 invoked by uid 99); 5 Oct 2015 10:36:21 -0000
Received: from mail-relay.apache.org (HELO mail-relay.apache.org)
 (140.211.11.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2015 10:36:21 +0000
Received: from s2laptop.dev.local
 (host81-156-40-151.range81-156.btcentralplus.com [81.156.40.151])
	by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with
 ESMTPSA id 9D78E1A0251
	for <users@tomcat.apache.org>; Mon,  5 Oct 2015 10:36:20 +0000 (UTC)
Subject: Re: Regarding StuckThreadDetectionValve
To: Tomcat Users List <users@tomcat.apache.org>
References: 
 <CALpj=OYDqx7H-FcAhJbY6pOHMTQj-AT5GJunbtPj7hDiE_h79A@mail.gmail.com>
 <5612298F.9020701@apache.org>
 <CALpj=OZuC3MS0gff69rd6B3RerNGOhHcwqtLM6LmOh1Z4C4mXg@mail.gmail.com>
 <56123936.5090009@apache.org>
 <CALpj=OZfjPZ8VWZ94qFTmvCiUvGeBkvpBa6h8fY47OHiMYfu9w@mail.gmail.com>
 <561240E4.6090306@apache.org>
 <CALpj=ObUm9jvAfnxzKU=cmhNTjS_Z-e_JmWW+=7RSBuz00SBew@mail.gmail.com>
From: Mark Thomas <markt@apache.org>
Message-ID: <561252A1.3060106@apache.org>
Date: Mon, 5 Oct 2015 11:36:17 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0)
 Gecko/20100101 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: 
 <CALpj=ObUm9jvAfnxzKU=cmhNTjS_Z-e_JmWW+=7RSBuz00SBew@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

On 05/10/2015 11:28, Yogesh Patel wrote:
> Hi Thomas , 
>                   Please see this image ...have a look at Time column

The list strips images. If you really want us to look at the image (not
that I think it will be remotely relevant) upload it somewhere and post
the URL (make sure it is publicly accessible and doe snot require any
form of registration to view it).

Mark


> 
> 
> ​
> 
> On 5 October 2015 at 14:50, Mark Thomas <markt@apache.org
> <mailto:markt@apache.org>> wrote:
> 
>     On 05/10/2015 10:09, Yogesh Patel wrote:
>     > Hi Thomas,
>     >
>     > Connector configuration is like :
>     > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" />
> 
>     That means you are using the BIO AJP connector.
> 
>     You don't have a problem with long running requests.
> 
>     You have a problem with thread starvation.
> 
>     AJP uses persistent connections.
> 
>     BIO requires one thread per connection, regardless of whether or nor
>     that connection is processing a request or waiting for the next one.
> 
>     httpd will (eventually, assuming mostly default config) create one AJP
>     connection for each httpd thread. If you have more httpd threads than
>     Tomcat threads you will eventually reach the point where httpd can't
>     create a Tomcat thread it requires and at that point it will appear
>     to hang.
> 
>     There are several possible fixes:
>     a) disable connection re-use in httpd for AJP connections
>     b) use the NIO AJP connector in Tomcat
>     c) increase maxThreads in Tomcat so it is > max threads in httpd
> 
>     For more explanation read this:
>     http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
>     <http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf>
> 
>     from slide 29 to 33
> 
>     Mark
> 
> 
> 
>     >
>     > On 5 October 2015 at 14:17, Mark Thomas <markt@apache.org
>     <mailto:markt@apache.org>> wrote:
>     >
>     >> On 05/10/2015 09:07, Yogesh Patel wrote:
>     >>> Thanks Mark Thomas ,
>     >>>
>     >>>    Our application is access by Apache TO Tomcat using AJP
>     Connector.
>     >>
>     >> OK. That answers one of the questions I asked. However, you haven't
>     >> provided the connector configuration.
>     >>
>     >>>     Problem :
>     >>>     Our application was mostly hanged,after looking at tomcat
>     manager it
>     >>> shown there are so many long running threads shown.
>     >>
>     >> The Tomcat Manager app does not show long running threads. It
>     does show
>     >> you how many requests are busy but again, a busy thread is not
>     >> necessarily processing a request.
>     >>
>     >>>     We want to recognize why so many threads are running since
>     long time.
>     >>
>     >> Threads are always "running". The key question is are those threads
>     >> 'idle' or 'busy'. An idle thread is waiting to be allocated work.
>     A busy
>     >> thread has been allocated work but the manager application can't
>     >> distinguish if that work is 'wait for a request to process' or
>     >> 'processing a request'.
>     >>
>     >>>     We want to detect such thread and want to kill these stucked
>     thread.
>     >>
>     >> You continue to make the (increasingly unlikely) assumption that 200
>     >> busy threads mean you have 200 threads processing long running
>     requests.
>     >> Had you provided the connector configuration I asked for (by that
>     I mean
>     >> the <Connector ... /> elements in server.xml) then I'd be in a better
>     >> position to tell you what is wrong.
>     >>
>     >> If you do have 200 long running requests then the
>     >> StuckThreadDetectionValve is how you detect them. That fact that that
>     >> valve is not reporting any long running requests should be a big clue
>     >> that your assumptions about what is going on are not correct.
>     >>
>     >> If, and it is a big if, you did have 200 concurrent long running
>     >> requests then there is no guaranteed way to stop them. If your
>     >> application checks for interrupts (most don't) then the
>     >> StuckThreadDetectionValve can interrupt them which should
>     terminate the
>     >> processing of that request but the time it would take to code the
>     >> application to handle that properly would normally be better spent
>     >> fixing the root cause of the long running request.
>     >>
>     >> Mark
>     >>
>     >>>
>     >>>
>     >>>
>     >>>
>     >>>
>     >>> On 5 October 2015 at 13:11, Mark Thomas <markt@apache.org
>     <mailto:markt@apache.org>> wrote:
>     >>>
>     >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
>     >>>>>         We are facing issues with long running thread in
>     tomcat . we
>     >> are
>     >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy threads
>     : 200,
>     >>>>> application  gets stucked due to these long running threads.
>     >>>>
>     >>>> What makes you think you have issues with long running threads?
>     A busy
>     >>>> thread isn't necessarily processing a request. Configuration errors
>     >>>> leading to thread starvation are a more typical cause of the
>     symptom you
>     >>>> describe rather than long running threads in the application.
>     >>>>
>     >>>>>          We implemented StuckThreadDetectionValve in
>     server.xml( <Valve
>     >>>>>
>     >>>>>   
>      className="org.apache.catalina.valves.StuckThreadDetectionValve"
>     >>>>>
>     >>>>>     threshold="60" />), but it  could not help out.
>     >>>>
>     >>>> Why not? Again, this suggests that long running threads aren't the
>     >> problem.
>     >>>>
>     >>>>>        So we implemented custom StuckThreadDetectionValve by
>     extending
>     >>>>> StuckThreadDetectionValve from
>     >>>>> catalina, it only goes to "constructor","initInternal",and in
>     >>>> "invoke"(when
>     >>>>> action fires), it does not go to function
>     "getStuckThreadNames()" even
>     >>>>> after threshold time.How to achieve the same.Is there any way
>     to detect
>     >>>>> stucked thread and kill them?
>     >>>>
>     >>>> First of all your invoke() method fails to call super.invoke()
>     so the
>     >>>> Valve is never going to do anything.
>     >>>>
>     >>>> Secondly, if the original valve didn't work adding a bunch of
>     >>>> System.out.println() lines isn't going to magically make it work.
>     >>>>
>     >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat
>     code so it
>     >>>> is no surprise that you never see a call to that method. That
>     method is
>     >>>> intended for use via JMX.
>     >>>>
>     >>>> I suggest you start again and tell us more about the problem
>     you are
>     >>>> trying to solve (lack of response) and your configuration. In
>     particular
>     >>>> we need to know your connector configuration and how users
>     access the
>     >>>> application. Do they connect directly to Tomcat or do they go via a
>     >>>> reverse proxy?
>     >>>>
>     >>>> Mark
>     >>>>
>     >>>>
>     >>>>>
>     >>>>>   My custom Valve is like following :
>     >>>>>
>     >>>>> public class StuckThreadDetection extends
>     StuckThreadDetectionValve
>     >>>>> {
>     >>>>> StuckThreadDetection stuckThreadDetection;
>     >>>>>
>     >>>>> public StuckThreadDetection()
>     >>>>> {
>     >>>>> System.out.println("in StuckThreadDetection constructor");
>     >>>>>
>     >>>>> }
>     >>>>>
>     >>>>> public void invoke(Request request, Response response) throws
>     >>>> IOException,
>     >>>>> ServletException
>     >>>>> {
>     >>>>> System.out.println("in invoke...");
>     >>>>>
>     >>>>> getNext().invoke(request, response);
>     >>>>> }
>     >>>>>
>     >>>>> @Override
>     >>>>> protected void initInternal() throws LifecycleException
>     >>>>> {
>     >>>>> System.out.println("In initInternal");
>     >>>>> super.initInternal();
>     >>>>> }
>     >>>>>
>     >>>>> @Override
>     >>>>> public String[] getStuckThreadNames()
>     >>>>> {
>     >>>>> System.out.println("in getStuckThreadNames...");
>     >>>>> String[] listStuckedThread = this.getStuckThreadNames();
>     >>>>>
>     >>>>> for (String threadName : listStuckedThread)
>     >>>>> {
>     >>>>> System.out.println(threadName);
>     >>>>> }
>     >>>>>
>     >>>>> return listStuckedThread;
>     >>>>>
>     >>>>> }
>     >>>>>
>     >>>>> @Override
>     >>>>> public String getInfo()
>     >>>>> {
>     >>>>> System.out.println("In getInfo");
>     >>>>> return super.getInfo();
>     >>>>> }
>     >>>>> }
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>
>     >>>>
>     >>>>
>     ---------------------------------------------------------------------
>     >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>     <mailto:users-unsubscribe@tomcat.apache.org>
>     >>>> For additional commands, e-mail: users-help@tomcat.apache.org
>     <mailto:users-help@tomcat.apache.org>
>     >>>>
>     >>>>
>     >>>
>     >>>
>     >>
>     >>
>     >> ---------------------------------------------------------------------
>     >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>     <mailto:users-unsubscribe@tomcat.apache.org>
>     >> For additional commands, e-mail: users-help@tomcat.apache.org
>     <mailto:users-help@tomcat.apache.org>
>     >>
>     >>
>     >
>     >
> 
> 
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>     <mailto:users-unsubscribe@tomcat.apache.org>
>     For additional commands, e-mail: users-help@tomcat.apache.org
>     <mailto:users-help@tomcat.apache.org>
> 
> 
> 
> 
> -- 
> /Thanks & Regards,/
> / /
> / Yogesh Patel/


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org