tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anthonyvie...@gmail.com
Subject Re: Tomcat dies suddenly
Date Fri, 12 Feb 2010 17:07:10 GMT
Is it possible to run this server with a basic tomcat application under load
to rule out the application causing the crash?

On Fri, Feb 12, 2010 at 4:20 AM, Carl <carl@etrak-plus.com> wrote:

> This problem continues to plague me.
>
> A quick recap so you don't have to search your memory or archives.
>
> The 10,000 foot view:  new Dell T105 and T110, Slackware 13.0 (64 bit),
> latest Java (64 bit) and latest Tomcat.  Machines only run Tomcat and a
> small, special purpose Java server (which I have also moved to another
> machine to make certain it wasn't causing any problems.)  Periodically,
> Tomcat just dies leaving no tracks in any log that I have been able to find.
> The application has run on a Slackware 12.1 (32 bit) for several years
> without problems (except for application bugs.)  I have run memTest86 for 30
> hours on the T110 with no problems reported.
>
> More details: the Dell 105 has an AMD processor and (currently) 8 GB
> memory. The T110 has a Xeon 3440 processor and 4 GB memory.  The current
> Java version is 1.6.0_18-b07.  The current Tomcat version is 6.0.24.
>
> The servers are lightly loaded with less than 100 sessions active at any
> one time.
>
> All of the following trials have produced the same results:
>
> 1.  Tried openSuse 64 bit.
>
> 2.  Tried 32 bit Slackware 13.
>
> 3.  Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB.
>
> 4.  Have fiddled with the JAVA_OPTS settings in catalina.sh.  The current
> settings are:
>
> JAVA_OPTS="-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/usr/local/tomcat/logs"
>
> I can see the incremental GC effects in both catalina.out and VisualJVM.
> Note the fairly small (512MB) heap but watching VisualJVM indicates this is
> sufficient (when a failure occurs, VisualJVM will report the last amount of
> memory used and this is always well under the max in both heap and permGen.)
>
> More information about the failures:
>
> 1.  They are clean kills as I can restart Tomcat immediately after failure
> and there is no port conflict.  As I understand it, this implies the linux
> process was killed (I have manually killed the java process with kill -9 and
> had the same result that I have observed when the system fails) or Tomcat
> was shut down normally, e.g., using shutdown.sh (this always leaves tracks
> in catalina.out and I am not seeing any so I do not believe this is the
> case.)
>
> 2.  They appear to be load related.  On heavy processing days, the system
> might fail every 15 minutes but it could also run for up to 10 days without
> failure but with lighter processing.  I have found a way to force a more
> frequent failure.  We have four war's deployed (I will call them A, B, C and
> D.)  They are all the same application but we use this process to enable
> access to different databases.  A user accesses the correct application by
> https://xx.com/A or B, etc.  A is used for production while the others
> have specific purposes.  Thus, A is always used while the others are used
> periodically.  If users start coming in on B, C and/or D, within hours the
> failure occurs (Tomcat shuts down bringing all of the users down, of
> course.)  Note that the failure still does not happen immediately.
>
> 3.  They do not appear to be caused by memory restrictions as 1) the old
> server had only 2 GB of memory and ran well, 2) I have tried adding memory
> to the new servers with no change in behavior and 3) the indications from
> top and the Slackware system monitor are that the system is not starved for
> memory.  In fact, yesterday, running on the T105 with 8 GB of memory, top
> never reported over 6 GB being used (0 swap being used) yet it failed at
> about 4:00PM.
>
> 4.  Most of the failures will occur after some amount of processing.  We
> update the war's and restart the Tomcats each morning at 1:00AM.  Most of
> the failures will occur toward the end of the day although heavy processing
> (or using multiple 'applications') may force it to happen earlier (the
> earliest failure has been around 1:00PM... it was the heaviest processing
> day ever.)  It is almost as if there is a bucket somewhere that gets filled
> up and, when filled, causes the failure.  (So there is no misunderstanding,
> there has never been an OOM condition reported anywhere that I can find.)
>
> Observations (or random musings):
>
> The fact that the failures occur after some amount of processing implies
> that the issue is related to memory usage, and, potentially, caused by a
> memory leak in the application.  However, 1) I have never seen (from
> VisualJVM) any issue with either heap or permGen and the incremental GC's
> reported in catalina.out look pretty normal and 2) top, vmstat, system
> monitor, etc. are not showing any issues with memory.
>
> The failures look a lot like the linux OOM killer (which Mark or Chris said
> way back at the beginning which is now 2-3 months ago.)   Does anyone have
> an idea where I could get information on tracking the linux signals that
> could cause this?
>
> Thanks,
>
> Carl
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message