tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Anderson <ranom...@gmail.com>
Subject Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Date Thu, 16 Jul 2015 16:04:32 GMT
Thanks, Dan.

This is not a problem.  The thread is in the pool waiting for work.  It's
what you'd normally see on a server that's not handling any requests.

It's awkward. Everything is going normal and suddenly, many threads start
waiting on the same monitor. After restart Tomcat, everything works without
even make any changes in Apache (service or config files).

 - When the problem occurs, check the access logs for both and see if the
request is making it from HTTPD to Tomcat.

OK. I'll check

   - Also when the problem occurs, try hitting Tomcat directly while the
problem is occurring.  See if you can access your app when you bypass HTTPD.

I've already tried this. Tomcat becomes very very very slow when it happens.

   - Post your HTTPD config

worker.properties:

worker.list=balancer

worker.tomsrv01.type=ajp13
worker.tomsrv01.host=x.y.z.1
worker.tomsrv01.port=8009
worker.tomsrv01.lbfactor=1
worker.tomsrv01.connect_timeout=10000
worker.tomsrv01.prepost_timeout=10000
worker.tomsrv01.socket_timeout=1800
worker.tomsrv01.connection_pool_timeout=60

worker.tomsrv02.type=ajp13
worker.tomsrv02.host=x.y.z.2
worker.tomsrv02.port=8009
worker.tomsrv02.lbfactor=1
worker.tomsrv02.connect_timeout=10000
worker.tomsrv02.prepost_timeout=10000
worker.tomsrv02.socket_timeout=1800
worker.tomsrv02.connection_pool_timeout=60

worker.tomsrv04.type=ajp13
worker.tomsrv04.host=x.y.z.3
worker.tomsrv04.port=8009
worker.tomsrv04.lbfactor=1
worker.tomsrv04.connect_timeout=10000
worker.tomsrv04.prepost_timeout=10000
worker.tomsrv04.socket_timeout=1800
worker.tomsrv04.connection_pool_timeout=60

worker.balancer.type=lb
worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04
worker.balancer.sticky_session=1

   - Make sure timeouts are configured appropriately  -

Ok


Do you have a firewall in between HTTPD & Tomcat?  Is there anything that
could be breaking the connection?

No.


Best regards.


2015-07-16 12:44 GMT-03:00 Daniel Mikusa <dmikusa@pivotal.io>:

> On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ranomail@gmail.com>
> wrote:
>
> > Hi,
> >
> > Sometimes, in moments of high traffic for our patterns (170 req/sec), we
> > have a lot of threads like that:
> >
> > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
> > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
> > [0x000000004ddcb000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000609a53198> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at
> >
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > at java.lang.Thread.run(Thread.java:745)
> >
>
> This is not a problem.  The thread is in the pool waiting for work.  It's
> what you'd normally see on a server that's not handling any requests.
>
>
> >
> > Thus, Tomcat hangs (all available threads stay in that state) and we have
> > to restart the service.
> >
> > Server version: Apache Tomcat/7.0.63
> > Server built:   Jun 30 2015 08:08:33 UTC
> > Server number:  7.0.63.0
> > OS Name:        Linux
> > OS Version:     2.6.18-194.17.1.el5
> > Architecture:   amd64
> > JVM Version:    1.7.0_80-b15
> > JVM Vendor:     Oracle Corporation
> >
> > We are using tomcat-native (APR connector) and Apache (2.2.3) as load
> > balancer with mod_jk.
> >
> >
> > server.xml
> >
> > <?xml version='1.0' encoding='utf-8'?>
> > <Server port="8005" shutdown="SHUTDOWN">
> >   <Listener className="org.apache.catalina.core.AprLifecycleListener"
> > SSLEngine="off" />
> >   <Listener className="org.apache.catalina.core.JasperListener" />
> >   <Listener
> > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> >   <Listener
> > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
> >   <Listener
> > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
> >   <GlobalNamingResources>
> >     <Resource name="UserDatabase" auth="Container"
> >               type="org.apache.catalina.UserDatabase"
> >               description="User database that can be updated and saved"
> >
>  factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> >               pathname="conf/tomcat-users.xml" />
> >   </GlobalNamingResources>
> >
> >   <Service name="Catalina">
> >
> >     <Connector port="8080" protocol="HTTP/1.1"
> >                connectionTimeout="20000"
> >                redirectPort="8443" />
> >     <Connector port="8009" enableLookups="false"
> connectionTimeout="60000"
> > protocol="AJP/1.3" redirectPort="8443" />
> >
> >     <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">
> >
> >       <Realm className="org.apache.catalina.realm.LockOutRealm">
> >         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> >                resourceName="UserDatabase" digest="SHA"/>
> >       </Realm>
> >
> >       <Host name="localhost"  appBase="webapps"
> >             unpackWARs="true" autoDeploy="true">
> >
> > <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> > threshold="60" interruptThreadThreshold="120" />
> >       </Host>
> >     </Engine>
> >   </Service>
> > </Server>
> >
> >
> > Any suggestions?
> >
>
> This could be an issue with HTTPD talking to Tomcat.
>    - When the problem occurs, check the access logs for both and see if the
> request is making it from HTTPD to Tomcat.
>    - Also when the problem occurs, try hitting Tomcat directly while the
> problem is occurring.  See if you can access your app when you bypass
> HTTPD.
>    - Post your HTTPD config
>    - Make sure timeouts are configured appropriately  -
> http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
>
> Do you have a firewall in between HTTPD & Tomcat?  Is there anything that
> could be breaking the connection?
>
> Dan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message