Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
MIME-Version: 1.0
In-Reply-To: 
 <CAGrg8DXad_Brc3dg66wYGs5b0tQh-2CdN_ZMd6_eGXUXLWK=AQ@mail.gmail.com>
References: 
 <CAGrg8DXad_Brc3dg66wYGs5b0tQh-2CdN_ZMd6_eGXUXLWK=AQ@mail.gmail.com>
Date: Thu, 16 Jul 2015 11:44:46 -0400
Message-ID: 
 <CAAqVxAzskkHEKRH_U4QZmi=EjY1EMBXewPOCzsvm1foegQD-kg@mail.gmail.com>
Subject: Re: Many Threads in TIMED_WAITING (parking) -
 [org.apache.tomcat.util.threads.TaskQueue]
From: Daniel Mikusa <dmikusa@pivotal.io>
To: Tomcat Users List <users@tomcat.apache.org>
Content-Type: multipart/alternative; boundary=001a11404440706da6051afff3f1

--001a11404440706da6051afff3f1
Content-Type: text/plain; charset=UTF-8

On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ranomail@gmail.com>
wrote:

> Hi,
>
> Sometimes, in moments of high traffic for our patterns (170 req/sec), we
> have a lot of threads like that:
>
> "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
> prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> [0x000000004ddcb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000609a53198> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
>

This is not a problem.  The thread is in the pool waiting for work.  It's
what you'd normally see on a server that's not handling any requests.


>
> Thus, Tomcat hangs (all available threads stay in that state) and we have
> to restart the service.
>
> Server version: Apache Tomcat/7.0.63
> Server built:   Jun 30 2015 08:08:33 UTC
> Server number:  7.0.63.0
> OS Name:        Linux
> OS Version:     2.6.18-194.17.1.el5
> Architecture:   amd64
> JVM Version:    1.7.0_80-b15
> JVM Vendor:     Oracle Corporation
>
> We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> balancer with mod_jk.
>
>
> server.xml
>
> <?xml version='1.0' encoding='utf-8'?>
> <Server port="8005" shutdown="SHUTDOWN">
>   <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="off" />
>   <Listener className="org.apache.catalina.core.JasperListener" />
>   <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
>   <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
>   <Listener
> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
>   <GlobalNamingResources>
>     <Resource name="UserDatabase" auth="Container"
>               type="org.apache.catalina.UserDatabase"
>               description="User database that can be updated and saved"
>               factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>               pathname="conf/tomcat-users.xml" />
>   </GlobalNamingResources>
>
>   <Service name="Catalina">
>
>     <Connector port="8080" protocol="HTTP/1.1"
>                connectionTimeout="20000"
>                redirectPort="8443" />
>     <Connector port="8009" enableLookups="false" connectionTimeout="60000"
> protocol="AJP/1.3" redirectPort="8443" />
>
>     <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
>
>       <Realm className="org.apache.catalina.realm.LockOutRealm">
>         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>                resourceName="UserDatabase" digest="SHA"/>
>       </Realm>
>
>       <Host name="localhost"  appBase="webapps"
>             unpackWARs="true" autoDeploy="true">
>
> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> threshold="60" interruptThreadThreshold="120" />
>       </Host>
>     </Engine>
>   </Service>
> </Server>
>
>
> Any suggestions?
>

This could be an issue with HTTPD talking to Tomcat.
   - When the problem occurs, check the access logs for both and see if the
request is making it from HTTPD to Tomcat.
   - Also when the problem occurs, try hitting Tomcat directly while the
problem is occurring.  See if you can access your app when you bypass HTTPD.
   - Post your HTTPD config
   - Make sure timeouts are configured appropriately  -
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html

Do you have a firewall in between HTTPD & Tomcat?  Is there anything that
could be breaking the connection?

Dan

--001a11404440706da6051afff3f1--