tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl" <c...@etrak-plus.com>
Subject Re: Tomcat dies suddenly
Date Sat, 30 Jan 2010 17:32:11 GMT
George,

Thanks for your reply.

I will check that IPV6 is disabled... can't remember for certain.

I have had the problem on both openSuse and Slackware which has lead me to 
believe the problem is in the JVM.  I guess I could try compiling the JVM 
from source on the target machine.

I ran memTest86 for 30 hours on the first server I encountered the problem 
with (the first server with Slackware and 64 bit java that I brought up) and 
nothing showed up.  I then, redid that machine with openSuse (thought the 
problem might be Slackware) but openSuse failed as quickly.

Thanks,

Carl

----- Original Message ----- 
From: "George Sexton" <georges@mhsoftware.com>
To: "'Tomcat Users List'" <users@tomcat.apache.org>
Sent: Saturday, January 30, 2010 11:46 AM
Subject: RE: Tomcat dies suddenly


> I've had this happen. Finally, I got a stack trace. In my case, there
> appears to be a bug in GLIBC, so when a reverse IP address lookup is done
> AND there is an IPV6 entry, it causes a problem.
>
> The solution appeared to be disabling IPV6 since I'm not using it. This is
> on OpenSuSE.
>
> It would be worth checking.
>
> You might also run MemTest86+ on it if you haven't already.
>
> George Sexton
> MH Software, Inc.
> http://www.mhsoftware.com/
> Voice: 303 438 9585
>
>
>> -----Original Message-----
>> From: Carl [mailto:carl@etrak-plus.com]
>> Sent: Friday, January 29, 2010 12:54 PM
>> To: Tomcat Users List
>> Subject: Tomcat dies suddenly
>>
>> 6-7 weeks ago, we built up some new servers and started having sudden
>> failures... Tomcat just stops with no error message, no system error
>> messages, nothing that I have been able to find so far.
>>
>> To refresh everyone's memory, this is a new server, a Dell T110 with a
>> Xeon 3440 processor and 4GB memory.  I have turned off both the turbo
>> mode and hyperthreading.
>>
>> The environment:
>>
>> 64 bit Slackware Linux
>>
>> java version "1.6.0_17"
>> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
>> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
>>
>> Tomcat: apache-tomcat-6.0.20
>>
>> These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m
>> -XX:MaxPermSize=368m"
>>
>> I have observed the memory usage and general performance with Java
>> VisualVM and have seen nothing strange.  I thought I was seeing GC as
>> memory usage was going up and down but in fact it was mostly people
>> coming onto the system and leaving it.  After several hours, the memory
>> settles to a baseline of about 375MB.  Forced GC never takes it below
>> that value and the ups and downs from the people coming onto and
>> leaving the system also returns it to pretty much that value.  The
>> maximum memory used never was above 700MB for the entire day.
>>
>> The server runs well, idling along at 2-5% load, except for a quick
>> spike during GC, serving jsp's, etc. at a reasonable speed.  Without
>> warning and with no tracks in any log (Tomcat or system) or to the
>> console, Tomcat just shuts down.  I can usually simply restart it as
>> the ports used by Tomcat are closed... today, I needed to run
>> shutdown.sh before I could run startup.sh (startup.sh gave no errors
>> but would not start Tomcat until I ran shutdown.sh and that process put
>> nothing in the logs... this is the first time this has happened.)
>>
>> Sometimes, the system will run for a week, sometimes for only several
>> hours, sometimes only for a few minutes.  Today, it ran until about
>> 1:00PM and has been down four times since then.
>>
>> The failure (Tomcat shutting down) is not always the same place in the
>> code (I have some debugging messages going to catalina.out.)
>>
>> Load does not seem to make a difference.
>>
>> I have tried another sever (Dell T105, AMD processor, 6GB memory) and
>> have observed the same results.   I have run memTest86 on the T110 for
>> about 30 hours and it showed nothing.
>>
>> I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it
>> lasted 15 minutes.  I have used the same server.xml on all the
>> installs:
>>
>> <Server port="8005" shutdown="SHUTDOWN">
>> <!--APR library loader. Documentation at /docs/apr.html -->
>>
>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>> SSLEngine="on" />
>>
>> <!--Initialize Jasper prior to webapps are loaded. Documentation at
>> /docs/jasper-howto.html -->
>>
>> <Listener className="org.apache.catalina.core.JasperListener" />
>>
>> <!-- JMX Support for the Tomcat server. Documentation at /docs/non-
>> existent.html -->
>>
>> <Listener
>> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
>>
>> <Listener
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> />
>>
>> <!-- Global JNDI resources
>>
>> Documentation at /docs/jndi-resources-howto.html
>>
>> -->
>>
>> <GlobalNamingResources>
>>
>> <!-- Editable user database that can also be used by
>>
>> UserDatabaseRealm to authenticate users
>>
>> -->
>>
>> <Resource name="UserDatabase" auth="Container"
>>
>> type="org.apache.catalina.UserDatabase"
>>
>> description="User database that can be updated and saved"
>>
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>
>> pathname="conf/tomcat-users.xml" />
>>
>> </GlobalNamingResources>
>>
>> <!-- A "Service" is a collection of one or more "Connectors" that share
>>
>> a single "Container" Note: A "Service" is not itself a "Container",
>>
>> so you may not define subcomponents such as "Valves" at this level.
>>
>> Documentation at /docs/config/service.html
>>
>> -->
>>
>> <Service name="Catalina">
>>
>>
>> <!--The connectors can use a shared executor, you can define one or
>> more named thread pools-->
>>
>> <!--
>>
>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>
>> maxThreads="150" minSpareThreads="4"/>
>>
>> -->
>>
>>
>>
>> <!-- A "Connector" represents an endpoint by which requests are
>> received
>>
>> and responses are returned. Documentation at :
>>
>> Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
>>
>> Java AJP Connector: /docs/config/ajp.html
>>
>> APR (HTTP/AJP) Connector: /docs/apr.html
>>
>> Define a non-SSL HTTP/1.1 Connector on port 8080
>>
>> -->
>>
>> <Connector port="8080" protocol="HTTP/1.1"
>>
>> maxHttpHeaderSize="8192"
>>
>> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>>
>> enableLookups="false" redirectPort="8443" scheme="http"
>> acceptCount="100"
>>
>> connectionTimeout="20000" disableUploadTimeout="true" />
>>
>> <!-- A "Connector" using the shared thread pool-->
>>
>> <!--
>>
>> <Connector executor="tomcatThreadPool"
>>
>> port="8080" protocol="HTTP/1.1"
>>
>> connectionTimeout="20000"
>>
>> redirectPort="8443" />
>>
>> -->
>>
>> <!-- Define a SSL HTTP/1.1 Connector on port 8443
>>
>> This connector uses the JSSE configuration, when using APR, the
>>
>> connector should be using the OpenSSL style configuration
>>
>> described in the APR documentation -->
>>
>>
>> <Connector port="8443" maxHttpHeaderSize="8192"
>>
>> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>>
>> enableLookups="false" disableUploadTimeout="true"
>>
>> acceptCount="100" scheme="https" secure="true"
>>
>> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
>>
>> keystoreFile="/usr/local/certs/tomcat_keystore.ks"
>> keystorePass="jellybean"/>
>>
>>
>> <!-- Define a SSL HTTP/1.1 Connector on port 443 -->
>>
>>
>> <Connector port="443" maxHttpHeaderSize="8192"
>>
>> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>>
>> enableLookups="false" disableUploadTimeout="true"
>>
>> acceptCount="100" scheme="https" secure="true"
>>
>> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
>>
>> keystoreFile="/usr/local/certs/tomcat_keystore.ks"
>> keystorePass="jellybean"/>
>>
>>
>> <!--
>>
>> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
>>
>> maxThreads="150" scheme="https" secure="true"
>>
>> clientAuth="false" sslProtocol="TLS" />
>>
>> -->
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>
>> <Connector port="8009"
>>
>> enableLookups="false" redirectPort="443" protocol="AJP/1.3" />
>>
>>
>>
>> <!-- An Engine represents the entry point (within Catalina) that
>> processes
>>
>> every request. The Engine implementation for Tomcat stand alone
>>
>> analyzes the HTTP headers included with the request, and passes them
>>
>> on to the appropriate Host (virtual host).
>>
>> Documentation at /docs/config/engine.html -->
>>
>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>
>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
>>
>> -->
>>
>> <Engine name="Catalina" defaultHost="localhost">
>>
>> <!--For clustering, please take a look at documentation at:
>>
>> /docs/cluster-howto.html (simple how to)
>>
>> /docs/config/cluster.html (reference documentation) -->
>>
>> <!--
>>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>
>> -->
>>
>> <!-- The request dumper valve dumps useful debugging information about
>>
>> the request and response data received and sent by Tomcat.
>>
>> Documentation at: /docs/config/valve.html -->
>>
>> <!--
>>
>> <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
>>
>> -->
>>
>> <!-- This Realm uses the UserDatabase configured in the global JNDI
>>
>> resources under the key "UserDatabase". Any edits
>>
>> that are performed against this UserDatabase are immediately
>>
>> available for use by the Realm. -->
>>
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>
>> resourceName="UserDatabase"/>
>>
>> <!-- Define the default virtual host
>>
>> Note: XML Schema validation will not work with Xerces 2.2.
>>
>> -->
>>
>> <Host name="localhost" appBase="webapps"
>>
>> unpackWARs="true" autoDeploy="true" deployOnStartup="true"
>>
>> xmlValidation="false" xmlNamespaceAware="false">
>>
>> <!-- SingleSignOn valve, share authentication between web applications
>>
>> Documentation at: /docs/config/valve.html -->
>>
>> <!--
>>
>> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
>>
>> -->
>>
>> <!-- Access log processes all example.
>>
>> Documentation at: /docs/config/valve.html -->
>>
>> <!--
>>
>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>> directory="logs"
>>
>> prefix="localhost_access_log." suffix=".txt" pattern="common"
>> resolveHosts="false"/>
>>
>> -->
>>
>> </Host>
>>
>> </Engine>
>>
>> </Service>
>>
>> </Server>
>>
>> When Tomcat shuts down, the memory that it was using seems to still be
>> held (as seen from top) but it is nowhere near the machine physical
>> memory.
>>
>> The application has been running on an older server (Dell 600SC, 32 bit
>> Slackware, 2GB memory) for several years and, while the application
>> will throw exceptions now and then, it never crashed.  This lead me to
>> believe the problem had something to do with the 64 bit JVM but, with
>> without seeing errors anywhere, I can't be certain and don't know what
>> I can do about it except go back to 32 bit.
>>
>> One time, I observed the heap and permGen memory usage with Visual JVM.
>> It was running around 600MB before I forced a GC and 375MB afterward.
>> Speed was good.  Memory usage from top was 2.4GB.  Five minutes later,
>> Tomcat stopped leaving no tracks that I could find.  The memory usage
>> from top was around 2.4GB.  The memory usage from Visual JVM was still
>> showing 400MB+ although the Tomcat process was gone.  I restarted
>> Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough
>> to close the ports (8080, 8443, 443.)  Tomcat stayed up for less than
>> an hour (under light load) and stopped again.  The memory used
>> according to top was less than 3GB but I didn't get the exact number.
>> I restarted it again (no server reboot) and it ran for the rest of the
>> night (light load) and top was showing 3.3GB for memory in the morning.
>>
>> Anyone have any ideas how I might track this problem down?
>>
>> Thanks,
>>
>> Carl
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message