tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Mixon" <rnmi...@qwest.net>
Subject Tomcat hangs with lots of DBCP "Object.wait" processes
Date Fri, 14 Oct 2005 06:27:29 GMT
I have a customer running Tomcat 4.0.18. About twice a week their system is
locking up - obviously we don't know why or I would not be asking.

Symptoms
1) CPU usage per TOP is not very high.
2) When requesting the main web page, the page goes white and never returns.

When this happened yesterday I was able to get a thread dump. It was quite
different than I've seen on this system before, almost 4000 lines long. What
was particularly noticable was that it had 74 threads that were almost
identically waiting (my interpretation, not sure its accurate, but it seems
to be saying so) in Commons DBCP objects. I've got an annotated (and
somewhat shortened) version of the thread dump below. 

Additional info:

Sun Java jdk1.5.0_04
Tomcat is of course fronted by Apache 2.0.48 and mod_jk.
CPU does not appear to be high, but the 
  # uname -a
  Linux acme2.acme.com 2.4.22-1.2115.nptlsmp #1 SMP Wed Oct 29 15:30:09 EST
2003 i686 i686 i386 GNU/Linux
I believe its Linux Fedora FC2, dual Xeon processors, 2GB RAM

Oh, the data source is defined with pretty much default parameters:
        <Context path="" docBase="httpdocs" reloadable="true" debug="0">
                <Resource name="jdbc/acme.com" auth="Container"
                        type="javax.sql.DataSource"/>
                <ResourceParams name="jdbc/acme.com">
                        <parameter>
                                <name>url</name>
 
<value>jdbc:mysql://localhost/acme?autoReconnect=true&amp;zeroDateTimeBehavi
or=convertToNull&amp;noDatetimeStringSync=true</value>
                        </parameter>
                        <parameter>
                                <name>driverClassName</name>
                                <value>com.mysql.jdbc.Driver</value>
                        </parameter>

                        <parameter>
                                <name>username</name>
                                <value>acme</value>
                        </parameter>
                        <parameter>
                                <name>password</name>
                                <value>acme</value>
                        </parameter>
                </ResourceParams>
        </Context>

Any ideas or assistance in tracking this down are appreciated

Thanks in advance - Richard 

----- Annotated stack trace follows ---

Full thread dump Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode):

"Thread-421" daemon prio=1 tid=0x085f8bc0 nid=0x686f in Object.wait()
[0x85826000..0x85827680]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49d33258> (a
org.apache.commons.dbcp.AbandonedObjectPool)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Unknown Source)
	- locked <0x49d33258> (a
org.apache.commons.dbcp.AbandonedObjectPool)
	at
org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool
.java:117)
	- locked <0x49d33258> (a
org.apache.commons.dbcp.AbandonedObjectPool)
	at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.ja
va:110)
	at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:3
12)
	at com.acme.Connector.CreateConnection(Connector.java:69)
	at com.acme.OpenMailData.insert(OpenMailData.java:282)
	at com.acme.trackopenmail.doPost(trackopenmail.java:48)
	at com.acme.trackopenmail.doGet(trackopenmail.java:19)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:247)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:193)
	at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:260)
	at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
	at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
	at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
	at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
	at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
	at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
	at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
	at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
	at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180
)
	at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
	at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.
java:170)
	at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
	at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172
)
	at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
	at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
	at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
	at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:174)
	at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
	at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
	at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
	at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
	at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261)
	at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:632)
	at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:590)
	at
org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:707)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:530)
	at java.lang.Thread.run(Thread.java:595)

  ... 51 more of these, though the actual application class varies between
two or three different classes.

"Thread-209" daemon prio=1 tid=0x086e0158 nid=0x7746 in Object.wait()
[0x7f701000..0x7f701500]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x4ab87400> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x4ab87400> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"Thread-208" daemon prio=1 tid=0x08607f00 nid=0x7745 in Object.wait()
[0x7f482000..0x7f482580]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x4ab873e0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x4ab873e0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"Thread-207" daemon prio=1 tid=0x087b2aa0 nid=0x7744 in Object.wait()
[0x7d4c2000..0x7d4c2600]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x4ab873c0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x4ab873c0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"Thread-206" daemon prio=1 tid=0x08677068 nid=0x7743 in Object.wait()
[0x7f680000..0x7f680680]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x4ab873a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x4ab873a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)29 
 
  ... 20 more of the DBCP waiting threads, followed by the rest of the
thread dump ...


"MonitorRunnable" daemon prio=1 tid=0x7c9ae6f0 nid=0x769e in Object.wait()
[0x7e449000..0x7e449680]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49d63f30> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
	at
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.jav
a:420)
	- locked <0x49d63f30> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
	at java.lang.Thread.run(Thread.java:595)

"Thread-5" daemon prio=1 tid=0x7c9acc08 nid=0x769d in Object.wait()
[0x7e3c8000..0x7e3c8700]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49d64888> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x49d64888> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"Thread-4" daemon prio=1 tid=0x7c9abff8 nid=0x769c in Object.wait()
[0x7e347000..0x7e347780]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49d64868> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x49d64868> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"Thread-3" daemon prio=1 tid=0x7c7ff3d8 nid=0x769b runnable
[0x7e2c6000..0x7e2c6800]
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked <0x49ca0038> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:450)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultSe
rverSocketFactory.java:107)
	at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java
:341)
	at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:497)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:530)
	at java.lang.Thread.run(Thread.java:595)

"Thread-2" daemon prio=1 tid=0x7c55fb88 nid=0x769a in Object.wait()
[0x7e245000..0x7e245480]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49d64828> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:509)
	- locked <0x49d64828> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"HostConfig[acmemarketing.com]" daemon prio=1 tid=0x7c55f798 nid=0x7699
waiting on condition [0x7e1c4000..0x7e1c4500]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[]" daemon prio=1 tid=0x7c1b0fe8 nid=0x7698 waiting on
condition [0x7e143000..0x7e143580]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"WebappLoader[]" daemon prio=1 tid=0x7c9991a8 nid=0x7697 waiting on
condition [0x7dae0000..0x7dae0600]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:1221)
	at
org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341)
	at java.lang.Thread.run(Thread.java:595)

"HostConfig[hospitalitytoolkitemail.com]" daemon prio=1 tid=0x7cae8e98
nid=0x7696 waiting on condition [0x7da09000..0x7da09680]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[]" daemon prio=1 tid=0x7be58710 nid=0x7695 waiting on
condition [0x7d7bb000..0x7d7bb700]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"WebappLoader[]" daemon prio=1 tid=0x7cc610e8 nid=0x7694 waiting on
condition [0x7d441000..0x7d441780]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:1221)
	at
org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341)
	at java.lang.Thread.run(Thread.java:595)

"HostConfig[localhost]" daemon prio=1 tid=0x7c8abba8 nid=0x7693 waiting on
condition [0x7d184000..0x7d184800]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985)
	at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[]" daemon prio=1 tid=0x7cae9890 nid=0x7692 waiting on
condition [0x7d103000..0x7d103480]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[/webdav]" daemon prio=1 tid=0x7bc62f30 nid=0x7691 waiting
on condition [0x7d082000..0x7d082500]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[/examples]" daemon prio=1 tid=0x088c70b8 nid=0x7690 waiting
on condition [0x7d001000..0x7d001580]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[/tomcat-docs]" daemon prio=1 tid=0x0820e688 nid=0x768f
waiting on condition [0x7cf80000..0x7cf80600]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[/pinEdit]" daemon prio=1 tid=0x0850be60 nid=0x768e waiting
on condition [0x7c4ff000..0x7c4ff680]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[/manager]" daemon prio=1 tid=0x08a42cf0 nid=0x768d waiting
on condition [0x7bf80000..0x7bf80700]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"StandardManager[/admin]" daemon prio=1 tid=0x0844dba8 nid=0x768c waiting on
condition [0x7bd80000..0x7bd80780]
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java
:780)
	at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
	at java.lang.Thread.run(Thread.java:595)

"Low Memory Detector" daemon prio=1 tid=0x081268e0 nid=0x7689 runnable
[0x00000000..0x00000000]

"CompilerThread1" daemon prio=1 tid=0x08125468 nid=0x7688 waiting on
condition [0x00000000..0x7a540718]

"CompilerThread0" daemon prio=1 tid=0x08124418 nid=0x7687 waiting on
condition [0x00000000..0x7a4bf798]

"AdapterThread" daemon prio=1 tid=0x081232e8 nid=0x7686 waiting on condition
[0x00000000..0x00000000]

"Signal Dispatcher" daemon prio=1 tid=0x08122468 nid=0x7685 waiting on
condition [0x00000000..0x00000000]

"Finalizer" daemon prio=1 tid=0x08117b98 nid=0x7684 in Object.wait()
[0x7a13d000..0x7a13d700]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49cea6d8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
	- locked <0x49cea6d8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x08116f20 nid=0x7683 in Object.wait()
[0x7a0bc000..0x7a0bc780]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x49cc2568> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:474)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x49cc2568> (a java.lang.ref.Reference$Lock)

"main" prio=1 tid=0x0805b220 nid=0x767c runnable [0xbfffc000..0xbfffd538]
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked <0x49d58e18> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:450)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at
org.apache.catalina.core.StandardServer.await(StandardServer.java:526)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:521)
	at org.apache.catalina.startup.Catalina.execute(Catalina.java:400)
	at org.apache.catalina.startup.Catalina.process(Catalina.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203)

"VM Thread" prio=1 tid=0x081142b8 nid=0x7682 runnable 

"GC task thread#0 (ParallelGC)" prio=1 tid=0x080e98b8 nid=0x767e runnable 

"GC task thread#1 (ParallelGC)" prio=1 tid=0x080e9ca0 nid=0x767f runnable 

"GC task thread#2 (ParallelGC)" prio=1 tid=0x080ea088 nid=0x7680 runnable 

"GC task thread#3 (ParallelGC)" prio=1 tid=0x080ea868 nid=0x7681 runnable 

"VM Periodic Task Thread" prio=1 tid=0x08127d70 nid=0x768a waiting on
condition 





---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message