tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vasily Ivanov" <basil...@gmail.com>
Subject Tomcat 4.1.13 dies every 3-4 days
Date Mon, 08 May 2006 02:45:00 GMT
Hello,

We are running J2EE web-application under Apache Server/Tomcat 4.1.31 on Linux.
At any point of time we've got around 250 active open sessions and
around 100 active threads running on server.
The problem is that tomcat occasionally gets dead every 3-4 days
without reporting any error to logs. There are 2Gb of memory allocated
to tomcat. And memory usage is normally 5-10%.

Some logs and server.xml attached.

I can not reproduce this in the dev environment.

Logs:

Here is the error from the mod_jk.log:
===================================================
[Mon May 08 10:25:14 2006] worker1 appl.company.com.au 0.004600
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.002706
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.013406
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.008658
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.002049
[Mon May 08 10:25:16 2006] worker1 appl.company.com.au 0.335327
[Mon May 08 10:25:16 2006] worker1 appl 0.009130
[Mon May 08 10:25:16 2006] worker1 appl.company.com.au 1.030997
[Mon May 08 10:25:16 2006] worker1 appl 0.008127
[Mon May 08 10:25:16 2006] worker1 appl 0.019742
[Mon May 08 10:25:16 2006] worker1 appl 0.007065
[Mon May 08 10:25:16 2006] worker1 appl.company.com.au 4.910941
[Mon May 08 10:25:17 2006] worker1 appl.company.com.au 6.059512
[Mon May 08 10:25:18 2006] [info] 
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:18 2006] [info]  ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 10.370808
[Mon May 08 10:25:18 2006] [info]  jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 2.020834
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 2.062616
[Mon May 08 10:25:18 2006] [info] 
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:18 2006] [info]  ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 7.267761
[Mon May 08 10:25:18 2006] [info]  jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:20 2006] worker1 appl.company.com.au 1.804931
[Mon May 08 10:25:20 2006] [info] 
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:20 2006] [info]  ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:20 2006] worker1 appl.company.com.au 5.307884
[Mon May 08 10:25:20 2006] [info]  jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:20 2006] worker1 appl.company.com.au 7.851162
[Mon May 08 10:25:21 2006] worker1 appl.company.com.au 0.229151
[Mon May 08 10:25:21 2006] [info] 
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:21 2006] [info]  ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:21 2006] worker1 appl.company.com.au 13.170979
[Mon May 08 10:25:21 2006] [info]  jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 6.087992
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 0.006206
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 11.215213
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 1.396763
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error] ajp_get_reply::jk_ajp_common.c
(1479): Tomcat is down or network problems. No response has been sent
to the client (yet)
[Mon May 08 10:25:26 2006] [info]  ajp_service::jk_ajp_common.c
(1696): Receiving from tomcat failed, recoverable operation attempt=0
[Mon May 08 10:25:26 2006] [info]  ajp_service::jk_ajp_common.c
(1724): Sending request to tomcat failed,  recoverable operation
attempt=1
[Mon May 08 10:25:26 2006] [info]  jk_open_socket::jk_connect.c (433):
connect to 127.0.0.1:8009 failed with errno=111
[Mon May 08 10:25:26 2006] [info] 
ajp_connect_to_endpoint::jk_ajp_common.c (877): Failed connecting to
tomcat. Tomcat is probably not started or is listening on the wrong
host/port (127.0.0.1:8009). Failed errno = 111
[Mon May 08 10:25:26 2006] [info]  ajp_send_request::jk_ajp_common.c
(1227): Error connecting to the Tomcat process.
[Mon May 08 10:25:26 2006] [info]  ajp_service::jk_ajp_common.c
(1724): Sending request to tomcat failed,  recoverable operation
attempt=2
[Mon May 08 10:25:26 2006] [info]  jk_open_socket::jk_connect.c (433):
connect to 127.0.0.1:8009 failed with errno=111
[Mon May 08 10:25:26 2006] [info] 
ajp_connect_to_endpoint::jk_ajp_common.c (877): Failed connecting to
tomcat. Tomcat is probably not started or is listening on the wrong
host/port (127.0.0.1:8009). Failed errno = 111
[Mon May 08 10:25:26 2006] [info]  ajp_send_request::jk_ajp_common.c
(1227): Error connecting to the Tomcat process.
[Mon May 08 10:25:26 2006] [info]  ajp_service::jk_ajp_common.c
(1724): Sending request to tomcat failed,  recoverable operation
attempt=3
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error] ajp_get_reply::jk_ajp_common.c
(1479): Tomcat is down or network problems. No response has been sent
to the client (yet)
===================================================

Here is the error_log from around this time:
===================================================
[Mon May 08 10:10:47 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=AE86ED2E4E74E1984F44046CA8D3A1B1,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:12:21 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=BFF2FDF5D0988AB6912B337DB852B9A0,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:12:21 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=BFF2FDF5D0988AB6912B337DB852B9A0,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:12:48 2006] [error] [client 161.117.83.199] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=A78A0AC23EB86064B5C9A725D77BE40B,
referer: http://appl/appl/authentication/authenticatestart.do
[Mon May 08 10:12:48 2006] [error] [client 161.117.83.199] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=A78A0AC23EB86064B5C9A725D77BE40B,
referer: http://appl/appl/authentication/authenticatestart.do
[Mon May 08 10:14:42 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=4337CBD952FBB9FC6D8607E9C5FDD02D,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:14:42 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=4337CBD952FBB9FC6D8607E9C5FDD02D,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:14:51 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=8F8A9C12B013F6BDF00E05EEEA89C42F,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
===================================================

Here is the tomcat process after restart:
===================================================
tomcat    9791     1 99 10:44 pts/2    00:01:03
/usr/j2sdk1.4.2_04/bin/java -server -Dappl_1 -Xms2048m -Xmx2048m
-Djava.endorsed.dirs=/app/tomcat/current/common/endorsed -classpath
/usr/j2sdk1.4.2_04/lib/tools.jar:/app/tomcat/current/bin/bootstrap.jar
-Dcatalina.base=/app/appl/tcinst1 -Dcatalina.home=/app/tomcat/curr
===================================================

Here is application log (not full) after restart:
===================================================
2006-05-08 10:44:06,857 [main] Http11Protocol INFO - Initializing
Coyote HTTP/1.1 on http-8029
Starting service Tomcat-Standalone
Apache Tomcat/4.1.31
...
8/05/2006 10:44:34 org.apache.commons.digester.Digester error
SEVERE: Parse Error at line 754 column 16: The content of element type
"definition" must match
"(icon?,display-name?,description?,put*,putList*)".
org.xml.sax.SAXParseException: The content of element type
"definition" must match
"(icon?,display-name?,description?,put*,putList*)".
	at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
Source)
	at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
	at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
	at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
	at org.apache.xerces.impl.dtd.XMLDTDValidator.handleEndElement(Unknown Source)
	at org.apache.xerces.impl.dtd.XMLDTDValidator.endElement(Unknown Source)
...
2006-05-08 10:44:35,672 [main] JkMain INFO - Jk running ID=0
time=17/90  config=/app/appl/tcinst1/conf/jk2.properties
...
2006-05-08 10:46:34,762 [TP-Processor23] Registry ERROR - Error
registering Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest25
javax.management.InstanceAlreadyExistsException:
Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest25
	at mx4j.server.MBeanServerImpl.register(MBeanServerImpl.java:1123)
	at mx4j.server.MBeanServerImpl.registerImpl(MBeanServerImpl.java:1054)
	at mx4j.server.MBeanServerImpl.registerMBeanImpl(MBeanServerImpl.java:1002)
	at mx4j.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:978)
	at org.apache.commons.modeler.Registry.registerComponent(Registry.java:871)
...
==================================================

This is server.xml
===================================================
<Server port="8005" shutdown="SHUTDOWN" debug="0">
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
            debug="0"/>
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
            debug="0"/>
  <GlobalNamingResources>
    <Environment name="simpleValue" type="java.lang.Integer" value="30"/>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
       description="User database that can be updated and saved">
    </Resource>
    <ResourceParams name="UserDatabase">
      <parameter>
        <name>factory</name>
        <value>org.apache.catalina.users.MemoryUserDatabaseFactory</value>
      </parameter>
      <parameter>
        <name>pathname</name>
        <value>conf/tomcat-users.xml</value>
      </parameter>
    </ResourceParams>
        <ResourceParams name="UserDatabase">
      <parameter>
        <name>factory</name>
        <value>org.apache.catalina.users.MemoryUserDatabaseFactory</value>
      </parameter>
      <parameter>
        <name>pathname</name>
        <value>conf/tomcat-users.xml</value>
      </parameter>
    </ResourceParams>
	<Resource name="jdbc/NonTransactionalDataSource"
        auth="Container"
        type="javax.sql.DataSource"/>
	<ResourceParams name="jdbc/NonTransactionalDataSource">
	<parameter>
       		<name>factory</name>
       		<value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
	</parameter>
	<parameter>
	    	<name>maxActive</name>
	    	<value>100</value>
	</parameter>
	<parameter>
    		<name>maxIdle</name>
    		<value>30</value>
	</parameter>
	<parameter>
    		<name>maxWait</name>
	    	<value>10000</value>
	</parameter>
	<parameter>
    		<name>validationQuery</name>
	    	<value>SELECT 1 FROM DUAL</value>
	</parameter>
	<parameter>
    		<name>username</name>
    		<value>appl_user</value>
	</parameter>
	<parameter>
    		<name>password</name>
    		<value>appl_user</value>
	</parameter>
	<parameter>
   		<name>driverClassName</name>
   		<value>oracle.jdbc.OracleDriver</value>
	</parameter>
	<parameter>
	        <name>url</name>
	        <value>jdbc:oracle:thin:@[...]</value>
	</parameter>
	</ResourceParams>
	<Resource name="jdbc/TransactionalDataSource"
        auth="Container"
        type="javax.sql.DataSource"/>

	<ResourceParams name="jdbc/TransactionalDataSource">
	<parameter>
       		<name>factory</name>
       		<value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
	</parameter>
	<parameter>
	    	<name>maxActive</name>
	    	<value>100</value>
	</parameter>
	<parameter>
    		<name>maxIdle</name>
    		<value>30</value>
	</parameter>
	<parameter>
    		<name>maxWait</name>
	    	<value>10000</value>
	</parameter>
	<parameter>
    		<name>validationQuery</name>
	    	<value>SELECT 1 FROM DUAL</value>
	</parameter>
	<parameter>
    		<name>username</name>
    		<value>appl_user</value>
	</parameter>
	<parameter>
    		<name>password</name>
    		<value>appl_user</value>
	</parameter>
	<parameter>
   		<name>driverClassName</name>
   		<value>oracle.jdbc.OracleDriver</value>
	</parameter>
	<parameter>
	        <name>url</name>
	        <value>jdbc:oracle:thin:@smedb501p:1521:wtop01</value>
	</parameter>
	</ResourceParams>
  </GlobalNamingResources>
  <Service name="Tomcat-Standalone">
    <Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
               port="8029" minProcessors="50" maxProcessors="300"
               enableLookups="true" redirectPort="8443"
               acceptCount="10" debug="0" connectionTimeout="20000"
               useURIValidationHack="false" disableUploadTimeout="true" />
    <Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
               port="8009" minProcessors="50" maxProcessors="300"
               enableLookups="false" redirectPort="8443"
               acceptCount="10" debug="0" connectionTimeout="0"
               useURIValidationHack="false"
               protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/>
    <Engine name="Standalone" defaultHost="localhost" debug="0">
      <Logger className="org.apache.catalina.logger.FileLogger"
              prefix="catalina_log." suffix=".txt"
              timestamp="true"/>
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
                 debug="0" resourceName="UserDatabase"/>
      <Host name="localhost" debug="0" appBase="webapps"
       unpackWARs="true" autoDeploy="true" liveDeploy="false">
        <Logger className="org.apache.catalina.logger.FileLogger"
                 directory="logs"  prefix="localhost_log." suffix=".txt"
	        timestamp="true"/>
		<Context path="/appl" reloadable="false" docBase="appl"  >
			<Logger className="org.apache.catalina.logger.SystemOutLogger"
verbosity="4" timestamp="true"/>
	        <ResourceLink global="jdbc/NonTransactionalDataSource"
name="jdbc/NonTransactionalDataSource" type="javax.sql.DataSource"/>
	        <ResourceLink global="jdbc/TransactionalDataSource"
name="jdbc/TransactionalDataSource" type="javax.sql.DataSource"/>
		</Context>
      </Host>
    </Engine>
  </Service>
</Server>
===================================================

Any help will be appreciated.
Thank you for your time.

Regards,
  Vasily

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message