tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl" <c...@etrak-plus.com>
Subject Re: Tomcat dies suddenly
Date Wed, 03 Feb 2010 21:37:24 GMT
George,

I did and found one instance but it is running on an applet on the client 
side so I don't think it should impact the server.

Thanks for the thoughts.

Carl

----- Original Message ----- 
From: "George Sexton" <georges@mhsoftware.com>
To: "'Tomcat Users List'" <users@tomcat.apache.org>
Sent: Wednesday, February 03, 2010 4:09 PM
Subject: RE: Tomcat dies suddenly


> You might want to search for getHostName by itself since it's an instance
> method and the variable name for the object would most certainly not be
> InetAddress.
>
> George Sexton
> MH Software, Inc.
> http://www.mhsoftware.com/
> Voice: 303 438 9585
>
>
>> -----Original Message-----
>> From: Carl [mailto:carl@etrak-plus.com]
>> Sent: Wednesday, February 03, 2010 12:55 PM
>> To: Tomcat Users List
>> Subject: Re: Tomcat dies suddenly
>>
>> George,
>>
>> I scanned the code for 'InetAddress.getHostName' and didn't find any
>> instances (that is what I thought but wanted to be certain.)
>>
>> It is frustrating how it will run very well for days and then suddenly
>> fail
>> every 15 minutes.  Last Friday was a 'fail every 15 minutes' day.  The
>> only
>> thing that was different last Friday was it was the last day of
>> registration
>> for several organizations and, while the total number of users was
>> still
>> around 80, the amount of work they were doing was substantially more
>> than
>> usual.  The only server I was using that day was a Dell T105.  When I
>> switched back to an older server (P4 dinosaur running 32 bit 5.5 java
>> and an
>> old Tomcat), it never failed.
>>
>> On the weekend, I was load testing both the rebuilt T110 and the T105
>> that
>> was running Friday and found that the T105 had zero swap although it
>> has 6GB
>> of memory and I don't think it ever got close to that amount used (I
>> also
>> now rebuilt that server.)  At any rate, no swap could, if I understand
>> it,
>> provoke the OOM killer which would give the type of failure we were
>> seeing.
>>
>> The load testing showed nothing.  I could overload the server, run it
>> out of
>> memory, etc. and it still behaved as expected (a little weird at times
>> but
>> never the type of failure we were seeing.)
>>
>> I have now added some java options:
>>
>> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -
>> XX:+PrintGCTimeStamps
>>  -XX:+HeapDumpOnOutOfMemoryError -
>> XX:HeapDumpPath=/usr/local/tomcat/logs
>>
>> and am monitoring it on a continual basis using VisualJVM (which shows
>> a lot
>> of information but has never shown anything at the time the system
>> fails.)
>> So far, the system is running great: quick response, memory in the 300-
>> 400MB
>> range, GC working as it should, plenty of free memory in the machine
>> (not
>> close to using swap) and no dying.
>>
>> If you can think of any rock I might look under, I would like to hear
>> about
>> it.
>>
>> TIA,
>>
>> Carl
>>
>>
>> ----- Original Message -----
>> From: "George Sexton" <georges@mhsoftware.com>
>> To: "'Tomcat Users List'" <users@tomcat.apache.org>
>> Sent: Wednesday, February 03, 2010 1:55 PM
>> Subject: RE: Tomcat dies suddenly
>>
>>
>> >> -----Original Message-----
>> >> From: Carl [mailto:carl@etrak-plus.com]
>> >> Sent: Saturday, January 30, 2010 10:32 AM
>> >> To: Tomcat Users List
>> >> Subject: Re: Tomcat dies suddenly
>> >>
>> >> George,
>> >>
>> >> Thanks for your reply.
>> >>
>> >> I will check that IPV6 is disabled... can't remember for certain.
>> >>
>> >> I have had the problem on both openSuse and Slackware which has lead
>> me
>> >> to
>> >> believe the problem is in the JVM.  I guess I could try compiling
>> the
>> >> JVM
>> >> from source on the target machine.
>> >
>> > Are you doing any kind of reverse DNS lookup? E.G.
>> > InetAddress.getHostName()?
>> >
>> > I've really seen this problem more than once with Linux. On both 32-
>> bit
>> > and
>> > 64-bit systems.
>> >
>> > What makes it crazy is that the system might run for weeks, but then
>> a
>> > certain user can trigger the error and it will happen twice in an
>> hour.
>> >
>> > If the defect is in GLIBC, it would happen on both distributions.
>> Really,
>> > if
>> > you're using InetAddress.getHostName(), this could be a problem.
>> >
>> >
>> >
>> >>
>> >> I ran memTest86 for 30 hours on the first server I encountered the
>> >> problem
>> >> with (the first server with Slackware and 64 bit java that I brought
>> >> up) and
>> >> nothing showed up.  I then, redid that machine with openSuse
>> (thought
>> >> the
>> >> problem might be Slackware) but openSuse failed as quickly.
>> >>
>> >> Thanks,
>> >>
>> >> Carl
>> >>
>> >> ----- Original Message -----
>> >> From: "George Sexton" <georges@mhsoftware.com>
>> >> To: "'Tomcat Users List'" <users@tomcat.apache.org>
>> >> Sent: Saturday, January 30, 2010 11:46 AM
>> >> Subject: RE: Tomcat dies suddenly
>> >>
>> >>
>> >> > I've had this happen. Finally, I got a stack trace. In my case,
>> there
>> >> > appears to be a bug in GLIBC, so when a reverse IP address lookup
>> is
>> >> done
>> >> > AND there is an IPV6 entry, it causes a problem.
>> >> >
>> >> > The solution appeared to be disabling IPV6 since I'm not using it.
>> >> This is
>> >> > on OpenSuSE.
>> >> >
>> >> > It would be worth checking.
>> >> >
>> >> > You might also run MemTest86+ on it if you haven't already.
>> >> >
>> >> > George Sexton
>> >> > MH Software, Inc.
>> >> > http://www.mhsoftware.com/
>> >> > Voice: 303 438 9585
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Carl [mailto:carl@etrak-plus.com]
>> >> >> Sent: Friday, January 29, 2010 12:54 PM
>> >> >> To: Tomcat Users List
>> >> >> Subject: Tomcat dies suddenly
>> >> >>
>> >> >> 6-7 weeks ago, we built up some new servers and started having
>> >> sudden
>> >> >> failures... Tomcat just stops with no error message, no system
>> error
>> >> >> messages, nothing that I have been able to find so far.
>> >> >>
>> >> >> To refresh everyone's memory, this is a new server, a Dell T110
>> with
>> >> a
>> >> >> Xeon 3440 processor and 4GB memory.  I have turned off both the
>> >> turbo
>> >> >> mode and hyperthreading.
>> >> >>
>> >> >> The environment:
>> >> >>
>> >> >> 64 bit Slackware Linux
>> >> >>
>> >> >> java version "1.6.0_17"
>> >> >> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
>> >> >> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
>> >> >>
>> >> >> Tomcat: apache-tomcat-6.0.20
>> >> >>
>> >> >> These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -
>> >> XX:PermSize=368m
>> >> >> -XX:MaxPermSize=368m"
>> >> >>
>> >> >> I have observed the memory usage and general performance with
>> Java
>> >> >> VisualVM and have seen nothing strange.  I thought I was seeing
>> GC
>> >> as
>> >> >> memory usage was going up and down but in fact it was mostly
>> people
>> >> >> coming onto the system and leaving it.  After several hours, the
>> >> memory
>> >> >> settles to a baseline of about 375MB.  Forced GC never takes it
>> >> below
>> >> >> that value and the ups and downs from the people coming onto and
>> >> >> leaving the system also returns it to pretty much that value.
>> The
>> >> >> maximum memory used never was above 700MB for the entire day.
>> >> >>
>> >> >> The server runs well, idling along at 2-5% load, except for a
>> quick
>> >> >> spike during GC, serving jsp's, etc. at a reasonable speed.
>> Without
>> >> >> warning and with no tracks in any log (Tomcat or system) or to
>> the
>> >> >> console, Tomcat just shuts down.  I can usually simply restart
it
>> as
>> >> >> the ports used by Tomcat are closed... today, I needed to run
>> >> >> shutdown.sh before I could run startup.sh (startup.sh gave no
>> errors
>> >> >> but would not start Tomcat until I ran shutdown.sh and that
>> process
>> >> put
>> >> >> nothing in the logs... this is the first time this has happened.)
>> >> >>
>> >> >> Sometimes, the system will run for a week, sometimes for only
>> >> several
>> >> >> hours, sometimes only for a few minutes.  Today, it ran until
>> about
>> >> >> 1:00PM and has been down four times since then.
>> >> >>
>> >> >> The failure (Tomcat shutting down) is not always the same place
>> in
>> >> the
>> >> >> code (I have some debugging messages going to catalina.out.)
>> >> >>
>> >> >> Load does not seem to make a difference.
>> >> >>
>> >> >> I have tried another sever (Dell T105, AMD processor, 6GB memory)
>> >> and
>> >> >> have observed the same results.   I have run memTest86 on the
>> T110
>> >> for
>> >> >> about 30 hours and it showed nothing.
>> >> >>
>> >> >> I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat
>> 6.0.24...
>> >> it
>> >> >> lasted 15 minutes.  I have used the same server.xml on all the
>> >> >> installs:
>> >> >>
>> >> >> <Server port="8005" shutdown="SHUTDOWN">
>> >> >> <!--APR library loader. Documentation at /docs/apr.html -->
>> >> >>
>> >> >> <Listener
>> className="org.apache.catalina.core.AprLifecycleListener"
>> >> >> SSLEngine="on" />
>> >> >>
>> >> >> <!--Initialize Jasper prior to webapps are loaded. Documentation
>> at
>> >> >> /docs/jasper-howto.html -->
>> >> >>
>> >> >> <Listener className="org.apache.catalina.core.JasperListener"
/>
>> >> >>
>> >> >> <!-- JMX Support for the Tomcat server. Documentation at
>> /docs/non-
>> >> >> existent.html -->
>> >> >>
>> >> >> <Listener
>> >> >> className="org.apache.catalina.mbeans.ServerLifecycleListener"
/>
>> >> >>
>> >> >> <Listener
>> >> >>
>> >>
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> >> >> />
>> >> >>
>> >> >> <!-- Global JNDI resources
>> >> >>
>> >> >> Documentation at /docs/jndi-resources-howto.html
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <GlobalNamingResources>
>> >> >>
>> >> >> <!-- Editable user database that can also be used by
>> >> >>
>> >> >> UserDatabaseRealm to authenticate users
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <Resource name="UserDatabase" auth="Container"
>> >> >>
>> >> >> type="org.apache.catalina.UserDatabase"
>> >> >>
>> >> >> description="User database that can be updated and saved"
>> >> >>
>> >> >> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>> >> >>
>> >> >> pathname="conf/tomcat-users.xml" />
>> >> >>
>> >> >> </GlobalNamingResources>
>> >> >>
>> >> >> <!-- A "Service" is a collection of one or more "Connectors"
that
>> >> share
>> >> >>
>> >> >> a single "Container" Note: A "Service" is not itself a
>> "Container",
>> >> >>
>> >> >> so you may not define subcomponents such as "Valves" at this
>> level.
>> >> >>
>> >> >> Documentation at /docs/config/service.html
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <Service name="Catalina">
>> >> >>
>> >> >>
>> >> >> <!--The connectors can use a shared executor, you can define
one
>> or
>> >> >> more named thread pools-->
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> >> >>
>> >> >> maxThreads="150" minSpareThreads="4"/>
>> >> >>
>> >> >> -->
>> >> >>
>> >> >>
>> >> >>
>> >> >> <!-- A "Connector" represents an endpoint by which requests
are
>> >> >> received
>> >> >>
>> >> >> and responses are returned. Documentation at :
>> >> >>
>> >> >> Java HTTP Connector: /docs/config/http.html (blocking & non-
>> >> blocking)
>> >> >>
>> >> >> Java AJP Connector: /docs/config/ajp.html
>> >> >>
>> >> >> APR (HTTP/AJP) Connector: /docs/apr.html
>> >> >>
>> >> >> Define a non-SSL HTTP/1.1 Connector on port 8080
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <Connector port="8080" protocol="HTTP/1.1"
>> >> >>
>> >> >> maxHttpHeaderSize="8192"
>> >> >>
>> >> >> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>> >> >>
>> >> >> enableLookups="false" redirectPort="8443" scheme="http"
>> >> >> acceptCount="100"
>> >> >>
>> >> >> connectionTimeout="20000" disableUploadTimeout="true" />
>> >> >>
>> >> >> <!-- A "Connector" using the shared thread pool-->
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Connector executor="tomcatThreadPool"
>> >> >>
>> >> >> port="8080" protocol="HTTP/1.1"
>> >> >>
>> >> >> connectionTimeout="20000"
>> >> >>
>> >> >> redirectPort="8443" />
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <!-- Define a SSL HTTP/1.1 Connector on port 8443
>> >> >>
>> >> >> This connector uses the JSSE configuration, when using APR, the
>> >> >>
>> >> >> connector should be using the OpenSSL style configuration
>> >> >>
>> >> >> described in the APR documentation -->
>> >> >>
>> >> >>
>> >> >> <Connector port="8443" maxHttpHeaderSize="8192"
>> >> >>
>> >> >> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>> >> >>
>> >> >> enableLookups="false" disableUploadTimeout="true"
>> >> >>
>> >> >> acceptCount="100" scheme="https" secure="true"
>> >> >>
>> >> >> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
>> >> >>
>> >> >> keystoreFile="/usr/local/certs/tomcat_keystore.ks"
>> >> >> keystorePass="jellybean"/>
>> >> >>
>> >> >>
>> >> >> <!-- Define a SSL HTTP/1.1 Connector on port 443 -->
>> >> >>
>> >> >>
>> >> >> <Connector port="443" maxHttpHeaderSize="8192"
>> >> >>
>> >> >> maxThreads="600" minSpareThreads="25" maxSpareThreads="75"
>> >> >>
>> >> >> enableLookups="false" disableUploadTimeout="true"
>> >> >>
>> >> >> acceptCount="100" scheme="https" secure="true"
>> >> >>
>> >> >> clientAuth="false" sslProtocol="TLS" SSLEnabled="true"
>> >> >>
>> >> >> keystoreFile="/usr/local/certs/tomcat_keystore.ks"
>> >> >> keystorePass="jellybean"/>
>> >> >>
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
>> >> >>
>> >> >> maxThreads="150" scheme="https" secure="true"
>> >> >>
>> >> >> clientAuth="false" sslProtocol="TLS" />
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> >> >>
>> >> >> <Connector port="8009"
>> >> >>
>> >> >> enableLookups="false" redirectPort="443" protocol="AJP/1.3" />
>> >> >>
>> >> >>
>> >> >>
>> >> >> <!-- An Engine represents the entry point (within Catalina)
that
>> >> >> processes
>> >> >>
>> >> >> every request. The Engine implementation for Tomcat stand alone
>> >> >>
>> >> >> analyzes the HTTP headers included with the request, and passes
>> them
>> >> >>
>> >> >> on to the appropriate Host (virtual host).
>> >> >>
>> >> >> Documentation at /docs/config/engine.html -->
>> >> >>
>> >> >> <!-- You should set jvmRoute to support load-balancing via AJP
ie
>> :
>> >> >>
>> >> >> <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <Engine name="Catalina" defaultHost="localhost">
>> >> >>
>> >> >> <!--For clustering, please take a look at documentation at:
>> >> >>
>> >> >> /docs/cluster-howto.html (simple how to)
>> >> >>
>> >> >> /docs/config/cluster.html (reference documentation) -->
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Cluster
>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <!-- The request dumper valve dumps useful debugging information
>> >> about
>> >> >>
>> >> >> the request and response data received and sent by Tomcat.
>> >> >>
>> >> >> Documentation at: /docs/config/valve.html -->
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Valve
>> className="org.apache.catalina.valves.RequestDumperValve"/>
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <!-- This Realm uses the UserDatabase configured in the global
>> JNDI
>> >> >>
>> >> >> resources under the key "UserDatabase". Any edits
>> >> >>
>> >> >> that are performed against this UserDatabase are immediately
>> >> >>
>> >> >> available for use by the Realm. -->
>> >> >>
>> >> >> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> >> >>
>> >> >> resourceName="UserDatabase"/>
>> >> >>
>> >> >> <!-- Define the default virtual host
>> >> >>
>> >> >> Note: XML Schema validation will not work with Xerces 2.2.
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <Host name="localhost" appBase="webapps"
>> >> >>
>> >> >> unpackWARs="true" autoDeploy="true" deployOnStartup="true"
>> >> >>
>> >> >> xmlValidation="false" xmlNamespaceAware="false">
>> >> >>
>> >> >> <!-- SingleSignOn valve, share authentication between web
>> >> applications
>> >> >>
>> >> >> Documentation at: /docs/config/valve.html -->
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Valve className="org.apache.catalina.authenticator.SingleSignOn"
>> />
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> <!-- Access log processes all example.
>> >> >>
>> >> >> Documentation at: /docs/config/valve.html -->
>> >> >>
>> >> >> <!--
>> >> >>
>> >> >> <Valve className="org.apache.catalina.valves.AccessLogValve"
>> >> >> directory="logs"
>> >> >>
>> >> >> prefix="localhost_access_log." suffix=".txt" pattern="common"
>> >> >> resolveHosts="false"/>
>> >> >>
>> >> >> -->
>> >> >>
>> >> >> </Host>
>> >> >>
>> >> >> </Engine>
>> >> >>
>> >> >> </Service>
>> >> >>
>> >> >> </Server>
>> >> >>
>> >> >> When Tomcat shuts down, the memory that it was using seems to
>> still
>> >> be
>> >> >> held (as seen from top) but it is nowhere near the machine
>> physical
>> >> >> memory.
>> >> >>
>> >> >> The application has been running on an older server (Dell 600SC,
>> 32
>> >> bit
>> >> >> Slackware, 2GB memory) for several years and, while the
>> application
>> >> >> will throw exceptions now and then, it never crashed.  This lead
>> me
>> >> to
>> >> >> believe the problem had something to do with the 64 bit JVM but,
>> >> with
>> >> >> without seeing errors anywhere, I can't be certain and don't know
>> >> what
>> >> >> I can do about it except go back to 32 bit.
>> >> >>
>> >> >> One time, I observed the heap and permGen memory usage with
>> Visual
>> >> JVM.
>> >> >> It was running around 600MB before I forced a GC and 375MB
>> >> afterward.
>> >> >> Speed was good.  Memory usage from top was 2.4GB.  Five minutes
>> >> later,
>> >> >> Tomcat stopped leaving no tracks that I could find.  The memory
>> >> usage
>> >> >> from top was around 2.4GB.  The memory usage from Visual JVM was
>> >> still
>> >> >> showing 400MB+ although the Tomcat process was gone.  I restarted
>> >> >> Tomcat (did not reboot) so Tomcat had been shutdown gracefully
>> >> enough
>> >> >> to close the ports (8080, 8443, 443.)  Tomcat stayed up for less
>> >> than
>> >> >> an hour (under light load) and stopped again.  The memory used
>> >> >> according to top was less than 3GB but I didn't get the exact
>> >> number.
>> >> >> I restarted it again (no server reboot) and it ran for the rest
>> of
>> >> the
>> >> >> night (light load) and top was showing 3.3GB for memory in the
>> >> morning.
>> >> >>
>> >> >> Anyone have any ideas how I might track this problem down?
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Carl
>> >> >
>> >> >
>> >> > ------------------------------------------------------------------
>> ---
>> >> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> > For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >
>> >> >
>> >>
>> >>
>> >> --------------------------------------------------------------------
>> -
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> > For additional commands, e-mail: users-help@tomcat.apache.org
>> >
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message