tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: Tomcat dies suddenly
Date Fri, 29 Jan 2010 20:18:45 GMT
On 29/01/2010 19:54, Carl wrote:
> 6-7 weeks ago, we built up some new servers and started having sudden failures... Tomcat
just stops with no error message, no system error messages, nothing that I have been able
to find so far.
> 
> To refresh everyone's memory, this is a new server, a Dell T110 with a Xeon 3440 processor
and 4GB memory.  I have turned off both the turbo mode and hyperthreading.
> 
> The environment:
> 
> 64 bit Slackware Linux
> 
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
> 
> Tomcat: apache-tomcat-6.0.20
> 
> These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m"
> 
> I have observed the memory usage and general performance with Java VisualVM and have
seen nothing strange.  I thought I was seeing GC as memory usage was going up and down but
in fact it was mostly people coming onto the system and leaving it.  After several hours,
the memory settles to a baseline of about 375MB.  Forced GC never takes it below that value
and the ups and downs from the people coming onto and leaving the system also returns it to
pretty much that value.  The maximum memory used never was above 700MB for the entire day.
> 
> The server runs well, idling along at 2-5% load, except for a quick spike during GC,
serving jsp's, etc. at a reasonable speed.  Without warning and with no tracks in any log
(Tomcat or system) or to the console, Tomcat just shuts down.  I can usually simply restart
it as the ports used by Tomcat are closed... today, I needed to run shutdown.sh before I could
run startup.sh (startup.sh gave no errors but would not start Tomcat until I ran shutdown.sh
and that process put nothing in the logs... this is the first time this has happened.)  
> 
> Sometimes, the system will run for a week, sometimes for only several hours, sometimes
only for a few minutes.  Today, it ran until about 1:00PM and has been down four times since
then.
> 
> The failure (Tomcat shutting down) is not always the same place in the code (I have some
debugging messages going to catalina.out.)
> 
> Load does not seem to make a difference.
> 
> I have tried another sever (Dell T105, AMD processor, 6GB memory) and have observed the
same results.   I have run memTest86 on the T110 for about 30 hours and it showed nothing.

> 
> I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it lasted 15 minutes.
 I have used the same server.xml on all the installs:
> 
> <Server port="8005" shutdown="SHUTDOWN">
> <!--APR library loader. Documentation at /docs/apr.html -->
> 
> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"
/>
> 
> <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html
-->
> 
> <Listener className="org.apache.catalina.core.JasperListener" />
> 
> <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
> 
> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
> 
> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
> 
> <!-- Global JNDI resources
> 
> Documentation at /docs/jndi-resources-howto.html
> 
> -->
> 
> <GlobalNamingResources>
> 
> <!-- Editable user database that can also be used by
> 
> UserDatabaseRealm to authenticate users
> 
> -->
> 
> <Resource name="UserDatabase" auth="Container"
> 
> type="org.apache.catalina.UserDatabase"
> 
> description="User database that can be updated and saved"
> 
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> 
> pathname="conf/tomcat-users.xml" />
> 
> </GlobalNamingResources>
> 
> <!-- A "Service" is a collection of one or more "Connectors" that share
> 
> a single "Container" Note: A "Service" is not itself a "Container", 
> 
> so you may not define subcomponents such as "Valves" at this level.
> 
> Documentation at /docs/config/service.html
> 
> -->
> 
> <Service name="Catalina">
> 
> 
> <!--The connectors can use a shared executor, you can define one or more named thread
pools-->
> 
> <!--
> 
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
> 
> maxThreads="150" minSpareThreads="4"/>
> 
> -->
> 
> 
> 
> <!-- A "Connector" represents an endpoint by which requests are received
> 
> and responses are returned. Documentation at :
> 
> Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
> 
> Java AJP Connector: /docs/config/ajp.html
> 
> APR (HTTP/AJP) Connector: /docs/apr.html
> 
> Define a non-SSL HTTP/1.1 Connector on port 8080
> 
> -->
> 
> <Connector port="8080" protocol="HTTP/1.1" 
> 
> maxHttpHeaderSize="8192"
> 
> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
> 
> enableLookups="false" redirectPort="8443" scheme="http" acceptCount="100"
> 
> connectionTimeout="20000" disableUploadTimeout="true" />
> 
> <!-- A "Connector" using the shared thread pool-->
> 
> <!--
> 
> <Connector executor="tomcatThreadPool"
> 
> port="8080" protocol="HTTP/1.1" 
> 
> connectionTimeout="20000" 
> 
> redirectPort="8443" />
> 
> --> 
> 
> <!-- Define a SSL HTTP/1.1 Connector on port 8443
> 
> This connector uses the JSSE configuration, when using APR, the 
> 
> connector should be using the OpenSSL style configuration
> 
> described in the APR documentation -->
> 
> 
> <Connector port="8443" maxHttpHeaderSize="8192"
> 
> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
> 
> enableLookups="false" disableUploadTimeout="true"
> 
> acceptCount="100" scheme="https" secure="true"
> 
> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
> 
> keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/>
> 
> 
> <!-- Define a SSL HTTP/1.1 Connector on port 443 -->
> 
> 
> <Connector port="443" maxHttpHeaderSize="8192"
> 
> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
> 
> enableLookups="false" disableUploadTimeout="true"
> 
> acceptCount="100" scheme="https" secure="true"
> 
> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
> 
> keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/>
> 
> 
> <!--
> 
> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
> 
> maxThreads="150" scheme="https" secure="true"
> 
> clientAuth="false" sslProtocol="TLS" />
> 
> -->
> 
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> 
> <Connector port="8009" 
> 
> enableLookups="false" redirectPort="443" protocol="AJP/1.3" />
> 
> 
> 
> <!-- An Engine represents the entry point (within Catalina) that processes
> 
> every request. The Engine implementation for Tomcat stand alone
> 
> analyzes the HTTP headers included with the request, and passes them
> 
> on to the appropriate Host (virtual host).
> 
> Documentation at /docs/config/engine.html -->
> 
> <!-- You should set jvmRoute to support load-balancing via AJP ie :
> 
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1"> 
> 
> --> 
> 
> <Engine name="Catalina" defaultHost="localhost">
> 
> <!--For clustering, please take a look at documentation at:
> 
> /docs/cluster-howto.html (simple how to)
> 
> /docs/config/cluster.html (reference documentation) -->
> 
> <!--
> 
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
> 
> --> 
> 
> <!-- The request dumper valve dumps useful debugging information about
> 
> the request and response data received and sent by Tomcat.
> 
> Documentation at: /docs/config/valve.html -->
> 
> <!--
> 
> <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
> 
> -->
> 
> <!-- This Realm uses the UserDatabase configured in the global JNDI
> 
> resources under the key "UserDatabase". Any edits
> 
> that are performed against this UserDatabase are immediately
> 
> available for use by the Realm. -->
> 
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> 
> resourceName="UserDatabase"/>
> 
> <!-- Define the default virtual host
> 
> Note: XML Schema validation will not work with Xerces 2.2.
> 
> -->
> 
> <Host name="localhost" appBase="webapps"
> 
> unpackWARs="true" autoDeploy="true" deployOnStartup="true"
> 
> xmlValidation="false" xmlNamespaceAware="false">
> 
> <!-- SingleSignOn valve, share authentication between web applications
> 
> Documentation at: /docs/config/valve.html -->
> 
> <!--
> 
> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
> 
> -->
> 
> <!-- Access log processes all example.
> 
> Documentation at: /docs/config/valve.html -->
> 
> <!--
> 
> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" 
> 
> prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
> 
> -->
> 
> </Host>
> 
> </Engine>
> 
> </Service>
> 
> </Server>
> 
> When Tomcat shuts down, the memory that it was using seems to still be held (as seen
from top) but it is nowhere near the machine physical memory.
> 
> The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB
memory) for several years and, while the application will throw exceptions now and then, it
never crashed.  This lead me to believe the problem had something to do with the 64 bit JVM
but, with without seeing errors anywhere, I can't be certain and don't know what I can do
about it except go back to 32 bit.

Given the circumstances, that seems like a reasonable next step.

> Anyone have any ideas how I might track this problem down?

It sounds like it is going to be trial an error.

The closest I have seen to this was a web application that hung on
Tomcat start. It appeared to be one or more gc bugs in the 64-bit JVM. I
had one set options gc options that worked on Solaris and failed on
linux and another set that worked on linux but failed on Solaris. I
never did find a set that worked on both.

When Tomcat shuts down, I assume there is no java process left running.
i.e. the process dies rather than hangs.

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message