tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rhonny David <rhonnyda...@yahoo.com>
Subject Tomcat Hangs with variable frequency
Date Wed, 22 Dec 2010 11:48:00 GMT
Hi ALL,



We are using tomcat 6.0 in production environment and facing a critical problem 
that tomcat hangs with variable frequency in a week. We have tried to figure out 
the problem but couldn't reach the root cause. To investigate the cause we used 
the lambda probe tool. Lamba probe shows multiple threads are blocked when 
tomcat hangs. Normally i have seen a similar type of stack trace for the thread 
which lambda probe shows as blocked, I am pasting it as follows:

net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket ( SharedSocket.java:693 ) 
net.sourceforge.jtds.jdbc.ResponseStream.getPacket ( ResponseStream.java:466 ) 
net.sourceforge.jtds.jdbc.ResponseStream.read ( ResponseStream.java:103 ) 
net.sourceforge.jtds.jdbc.TdsCore.nextToken ( TdsCore.java:2202 ) 
net.sourceforge.jtds.jdbc.TdsCore.getNextRow ( TdsCore.java:764 ) 
net.sourceforge.jtds.jdbc.JtdsResultSet.next ( JtdsResultSet.java:593 ) 
net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.close ( 
DelegatingResultSet.java:187 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close ( 
DelegatingStatement.java:163 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.passivate ( 
DelegatingConnection.java:426 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close ( 
DelegatingConnection.java:246 ) 
org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose ( 
PoolableConnection.java:122 ) 
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject ( 
PoolableConnectionFactory.java:628 ) 
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.invalidateObject ( 
GenericObjectPool.java:1247 ) 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.invalidateObject ( 
AbandonedObjectPool.java:125 ) 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.removeAbandoned ( 
AbandonedObjectPool.java:158 ) 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject ( 
AbandonedObjectPool.java:77 ) 
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection ( 
PoolingDataSource.java:106 ) 
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection ( 
BasicDataSource.java:1044 ) 
com.avanza.contactcenter.dbutil.DBConnection.getConnection ( 
DBConnection.java:154 ) 
com.avanza.contactcenter.db.TableFieldsMappingProcessor.getDynamicColumnsReportList
 ( TableFieldsMappingProcessor.java:696 ) 
com.avanza.contactcenter.processor.CustomerDetailActionProcessor.CustomerDetailInfoSearch
 ( CustomerDetailActionProcessor.java:187 ) 
com.avanza.contactcenter.processor.RdvInboundActionProcessor.responseCustomerDetailInfo
 ( RdvInboundActionProcessor.java:817 ) 
com.avanza.contactcenter.integration.RendezvousMessageSource.processIncomingMessage
 ( RendezvousMessageSource.java:431 ) 
com.avanza.contactcenter.integration.RendezvousMessageSource.run ( 
RendezvousMessageSource.java:136 )


As we have been investigating the issue from a month, so i would like to share 
it with you though you can understand the scenario in depth. Initially we 
thought that there are connection leak problems in our application because the 
lambda probe showed us that db connections are continuously busy. We scanned 
 twice the DAO layer of our application to search any connection leaks, but we 
didn't find any. At that time our application's Context.xml  file didn't contain 
the removeAbandoned property. After DAO layer scan, we decide to set 
the removeAbandoned = true, so that connection pool refreshes at run time. But 
this also didn't work for us. Lambda probe still showed us blocked threads when 
tomcat hanged, but yes by setting the removeAbandoned = true connection pool 
continued to refresh.

When tomcat hanged today we noticed following statistics on lambda probe, (along 
with blocked threads for which stack trace is already provided above):



System OVerView

Memory utilization
 -- Free: 469.95 MB Total: 595.5 MB Max: 1,016.12 MB 


OS information
JVM:  Java(TM) 2 Runtime Environment, Standard Edition  1.5.0_10-b03  Java 
HotSpot(TM) Client VM) 
OS: Windows 2003 (Service Pack 2) x86 5.2
Processors: 8
Current time: Wed Dec 22 13:00:48 GMT+05:00 2010
Working dir: C:\Program Files\Apache Software Foundation\Tomcat 6.0


Current memory usage

                                 Used     Committed   Maximum    Initial  Group
                                 
Survivor Space:       1.26Mb   4.63Mb      7.88Mb     4.56Mb    HEAP 

Perm Gen:               34.28Mb  34.50Mb     64.00Mb    8.00Mb   NON_HEAP 
  
Tenured Gen:          115.20Mb  553.88Mb  945.25Mb  553.88Mb   HEAP 


Eden Space:             2.08Mb   37.00Mb    63.00Mb    37.00Mb   HEAP 


Code Cache            12.37Mb  12.47Mb  32.00Mb  192.00Kb  NON_HEAP 


Total                        165.65Mb  642.47Mb  1.09Gb  603.63Mb  TOTAL 


OS information
OS Name: Windows 2003 OS Version: 5.2 Total RAM: 2.00Gb Free 
RAM: 2.00Gb Committed JVM memory: 681.23Mb Total swap: 4.00Gb Free swap: 4.00Gb 

(In above line,Free RAM is 2GB, it might be due to that operating system is 32 
bit and hardware is also 32 bit)

STATUS:

ajp-8009

Current thread count: 0 Current threads busy: 0 Max threads: 200 Max spare 
threads: 0 Min spare threads: 0 
Max time (ms): 0 Processing time (ms): 0 Request count: 0 Error count: 0 
Received: 0b Sent: 0b 
No processors 
http-8080



Current thread count: 60 Current threads busy: 5 Max threads: 200 Max spare 
threads: 0 Min spare threads: 0 
Max time (ms): 430500 Processing time (ms): 133505594 Request count: 390058 
Error count: 8904 Received: 24Mb Sent: 4Gb 


Context.xml

<Resource factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory" 
auth="Application" removeAbandoned="true" logAbandoned="true" 
validationQuery="select 1" driverClassName="net.sourceforge.jtds.jdbc.Driver" 
maxActive="60" maxIdle="20" maxWait="10000" name="DB_Pool" 
type="javax.sql.DataSource"  url="jdbc:jtds:sqlserver://10.22.4.29/XYZ" />



Note that we have noticed multiple times that there are no deadlocks at database 
end when tomcat crashed and memory and CPU utilization of application server 
machine and database machine is always normal. We are using SQL Server 2005 
Enterprise in production environment. The number of concurrent users to use the 
application simultaneously is 50. 



Will appreciate if any one can give the idea by looking at above statistics that 
what is area of problem. Will appreciate.



Regards,
David


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message