tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl" <c...@etrak-plus.com>
Subject Re: Tomcat dies suddenly
Date Fri, 29 Jan 2010 20:36:41 GMT
Mark,

Thanks for your quick reply.

Can I run the 32 bit JVM on the 64 bit linux (I think I can but just wanted 
to confirm)?

Thanks,

Carl
----- Original Message ----- 
From: "Mark Thomas" <markt@apache.org>
To: "Tomcat Users List" <users@tomcat.apache.org>
Sent: Friday, January 29, 2010 3:18 PM
Subject: Re: Tomcat dies suddenly


> On 29/01/2010 19:54, Carl wrote:
>> 6-7 weeks ago, we built up some new servers and started having sudden 
>> failures... Tomcat just stops with no error message, no system error 
>> messages, nothing that I have been able to find so far.
>>
>> To refresh everyone's memory, this is a new server, a Dell T110 with a 
>> Xeon 3440 processor and 4GB memory.  I have turned off both the turbo 
>> mode and hyperthreading.
>>
>> The environment:
>>
>> 64 bit Slackware Linux
>>
>> java version "1.6.0_17"
>> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
>> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
>>
>> Tomcat: apache-tomcat-6.0.20
>>
>> These are the current 
>> JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m"
>>
>> I have observed the memory usage and general performance with Java 
>> VisualVM and have seen nothing strange.  I thought I was seeing GC as 
>> memory usage was going up and down but in fact it was mostly people 
>> coming onto the system and leaving it.  After several hours, the memory 
>> settles to a baseline of about 375MB.  Forced GC never takes it below 
>> that value and the ups and downs from the people coming onto and leaving 
>> the system also returns it to pretty much that value.  The maximum memory 
>> used never was above 700MB for the entire day.
>>
>> The server runs well, idling along at 2-5% load, except for a quick spike 
>> during GC, serving jsp's, etc. at a reasonable speed.  Without warning 
>> and with no tracks in any log (Tomcat or system) or to the console, 
>> Tomcat just shuts down.  I can usually simply restart it as the ports 
>> used by Tomcat are closed... today, I needed to run shutdown.sh before I 
>> could run startup.sh (startup.sh gave no errors but would not start 
>> Tomcat until I ran shutdown.sh and that process put nothing in the 
>> logs... this is the first time this has happened.)
>>
>> Sometimes, the system will run for a week, sometimes for only several 
>> hours, sometimes only for a few minutes.  Today, it ran until about 
>> 1:00PM and has been down four times since then.
>>
>> The failure (Tomcat shutting down) is not always the same place in the 
>> code (I have some debugging messages going to catalina.out.)
>>
>> Load does not seem to make a difference.
>>
>> I have tried another sever (Dell T105, AMD processor, 6GB memory) and 
>> have observed the same results.   I have run memTest86 on the T110 for 
>> about 30 hours and it showed nothing.
>>
>> I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it 
>> lasted 15 minutes.  I have used the same server.xml on all the installs:
>>
>> <Server port="8005" shutdown="SHUTDOWN">
>> <!--APR library loader. Documentation at /docs/apr.html -->
>>
>> <Listener className="org.apache.catalina.core.AprLifecycleListener" 
>> SSLEngine="on" />
>>
>> <!--Initialize Jasper prior to webapps are loaded. Documentation at 
>> /docs/jasper-howto.html -->
>>
>> <Listener className="org.apache.catalina.core.JasperListener" />
>>
>> <!-- JMX Support for the Tomcat server. Documentation at 
>> /docs/non-existent.html -->
>>
>> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" 
>> />
>>
>> <Listener 
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" 
>> />
>>
>> <!-- Global JNDI resources
>>
>> Documentation at /docs/jndi-resources-howto.html
>>
>> -->
>>
>> <GlobalNamingResources>
>>
>> <!-- Editable user database that can also be used by
>>
>> UserDatabaseRealm to authenticate users
>>
>> -->
>>
>> <Resource name="UserDatabase" auth="Container"
>>
>> type="org.apache.catalina.UserDatabase"
>>
>> description="User database that can be updated and saved"
>>
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>
>> pathname="conf/tomcat-users.xml" />
>>
>> </GlobalNamingResources>
>>
>> <!-- A "Service" is a collection of one or more "Connectors" that share
>>
>> a single "Container" Note: A "Service" is not itself a "Container",
>>
>> so you may not define subcomponents such as "Valves" at this level.
>>
>> Documentation at /docs/config/service.html
>>
>> -->
>>
>> <Service name="Catalina">
>>
>>
>> <!--The connectors can use a shared executor, you can define one or more 
>> named thread pools-->
>>
>> <!--
>>
>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>
>> maxThreads="150" minSpareThreads="4"/>
>>
>> -->
>>
>>
>>
>> <!-- A "Connector" represents an endpoint by which requests are received
>>
>> and responses are returned. Documentation at :
>>
>> Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
>>
>> Java AJP Connector: /docs/config/ajp.html
>>
>> APR (HTTP/AJP) Connector: /docs/apr.html
>>
>> Define a non-SSL HTTP/1.1 Connector on port 8080
>>
>> -->
>>
>> <Connector port="8080" protocol="HTTP/1.1"
>>
>> maxHttpHeaderSize="8192"
>>
>> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>>
>> enableLookups="false" redirectPort="8443" scheme="http" acceptCount="100"
>>
>> connectionTimeout="20000" disableUploadTimeout="true" />
>>
>> <!-- A "Connector" using the shared thread pool-->
>>
>> <!--
>>
>> <Connector executor="tomcatThreadPool"
>>
>> port="8080" protocol="HTTP/1.1"
>>
>> connectionTimeout="20000"
>>
>> redirectPort="8443" />
>>
>> -->
>>
>> <!-- Define a SSL HTTP/1.1 Connector on port 8443
>>
>> This connector uses the JSSE configuration, when using APR, the
>>
>> connector should be using the OpenSSL style configuration
>>
>> described in the APR documentation -->
>>
>>
>> <Connector port="8443" maxHttpHeaderSize="8192"
>>
>> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>>
>> enableLookups="false" disableUploadTimeout="true"
>>
>> acceptCount="100" scheme="https" secure="true"
>>
>> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
>>
>> keystoreFile="/usr/local/certs/tomcat_keystore.ks" 
>> keystorePass="jellybean"/>
>>
>>
>> <!-- Define a SSL HTTP/1.1 Connector on port 443 -->
>>
>>
>> <Connector port="443" maxHttpHeaderSize="8192"
>>
>> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>>
>> enableLookups="false" disableUploadTimeout="true"
>>
>> acceptCount="100" scheme="https" secure="true"
>>
>> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
>>
>> keystoreFile="/usr/local/certs/tomcat_keystore.ks" 
>> keystorePass="jellybean"/>
>>
>>
>> <!--
>>
>> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
>>
>> maxThreads="150" scheme="https" secure="true"
>>
>> clientAuth="false" sslProtocol="TLS" />
>>
>> -->
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>
>> <Connector port="8009"
>>
>> enableLookups="false" redirectPort="443" protocol="AJP/1.3" />
>>
>>
>>
>> <!-- An Engine represents the entry point (within Catalina) that 
>> processes
>>
>> every request. The Engine implementation for Tomcat stand alone
>>
>> analyzes the HTTP headers included with the request, and passes them
>>
>> on to the appropriate Host (virtual host).
>>
>> Documentation at /docs/config/engine.html -->
>>
>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>
>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
>>
>> -->
>>
>> <Engine name="Catalina" defaultHost="localhost">
>>
>> <!--For clustering, please take a look at documentation at:
>>
>> /docs/cluster-howto.html (simple how to)
>>
>> /docs/config/cluster.html (reference documentation) -->
>>
>> <!--
>>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>
>> -->
>>
>> <!-- The request dumper valve dumps useful debugging information about
>>
>> the request and response data received and sent by Tomcat.
>>
>> Documentation at: /docs/config/valve.html -->
>>
>> <!--
>>
>> <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
>>
>> -->
>>
>> <!-- This Realm uses the UserDatabase configured in the global JNDI
>>
>> resources under the key "UserDatabase". Any edits
>>
>> that are performed against this UserDatabase are immediately
>>
>> available for use by the Realm. -->
>>
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>
>> resourceName="UserDatabase"/>
>>
>> <!-- Define the default virtual host
>>
>> Note: XML Schema validation will not work with Xerces 2.2.
>>
>> -->
>>
>> <Host name="localhost" appBase="webapps"
>>
>> unpackWARs="true" autoDeploy="true" deployOnStartup="true"
>>
>> xmlValidation="false" xmlNamespaceAware="false">
>>
>> <!-- SingleSignOn valve, share authentication between web applications
>>
>> Documentation at: /docs/config/valve.html -->
>>
>> <!--
>>
>> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
>>
>> -->
>>
>> <!-- Access log processes all example.
>>
>> Documentation at: /docs/config/valve.html -->
>>
>> <!--
>>
>> <Valve className="org.apache.catalina.valves.AccessLogValve" 
>> directory="logs"
>>
>> prefix="localhost_access_log." suffix=".txt" pattern="common" 
>> resolveHosts="false"/>
>>
>> -->
>>
>> </Host>
>>
>> </Engine>
>>
>> </Service>
>>
>> </Server>
>>
>> When Tomcat shuts down, the memory that it was using seems to still be 
>> held (as seen from top) but it is nowhere near the machine physical 
>> memory.
>>
>> The application has been running on an older server (Dell 600SC, 32 bit 
>> Slackware, 2GB memory) for several years and, while the application will 
>> throw exceptions now and then, it never crashed.  This lead me to believe 
>> the problem had something to do with the 64 bit JVM but, with without 
>> seeing errors anywhere, I can't be certain and don't know what I can do 
>> about it except go back to 32 bit.
>
> Given the circumstances, that seems like a reasonable next step.
>
>> Anyone have any ideas how I might track this problem down?
>
> It sounds like it is going to be trial an error.
>
> The closest I have seen to this was a web application that hung on
> Tomcat start. It appeared to be one or more gc bugs in the 64-bit JVM. I
> had one set options gc options that worked on Solaris and failed on
> linux and another set that worked on linux but failed on Solaris. I
> never did find a set that worked on both.
>
> When Tomcat shuts down, I assume there is no java process left running.
> i.e. the process dies rather than hangs.
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message