tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl" <>
Subject Re: Tomcat dies suddenly
Date Sat, 06 Feb 2010 00:45:38 GMT

I have yet to do any sampling, I have been visually monitoring the servers 
with the VisualJVM, the Slackware system monitor, top and vmstat.  I was 
using JMeter to stress test the application.

The old server is Slackware 12.1 (32 bit), Tomcat 5.5, Java 1.5.0_01.

The new servers are Slackware 13 (64 bit), Tomcat 6.0.24, Java 1.6_18.

The old server has run flawlessly for several years.  The new servers are 
kicking my butt:

- They will run for a period of time (could be 10 days, could be 15 minutes) 
and silently fail, no error logs, no memory dump, nothing in catalina.out, 
no tracks that I have been able to find.

- The issue seems load related and cumulative.  A week ago was a very heavy 
processing day and the server failed repeatedly starting about noon. 
Otherwise, it would usually fail late in the day.

-  I have tried different heap sizes.  My first inclination was a larger 
heap but, as Chuck pointed out, that was probably the wrong thing.  I am 
currently running with a 500MB heap (384MB permGen.)  Been up for two days 
and looks great.

-  I have quite a bit of logging to catalina.out because I was, at one 
point, looking for a connection leak.  I also started incremental GC and 
printing the GC instances using the appropriate java flags.  The last entry 
for the last failure was a GC.

-  I tried provoking the failure by stressing the application using 
JMeter... wouldn't fail even when I forced an OOM.

Your idea of sampling is interesting... may try that next.



----- Original Message ----- 
From: "Jonathan Mast" <>
To: "Tomcat Users List" <>
Sent: Friday, February 05, 2010 7:11 PM
Subject: Re: Tomcat dies suddenly

> Hi Carl, I've skimmed thru all your posts, please excuse me if these
> questions are redundant.
> You've mentioned VisualJVM, JMeter, and Slackware monitoring tools, it 
> seems
> to me that you are using these to occasionally to monitor you Tomcat
> instance, when things seem to be going badly (thats my perception from
> reading your posts)
> Have you done any long term sampling (Tomcat threads, memory, etc) of your
> system?
> I use Slackware 12.2 (32-bit), Httpd 2.2, Java 1.6, Tomcat 6.0.18 and had
> Tomcat die in a manner similiar to yours:  First it died silently (nothing
> in the catalina.out), the next time (about a week later) it became
> non-responsive complaining about PermGen, OOMException and so on.  I 
> haven't
> determined the cause yet, I rolled over to another server (exact same
> hardware/software btw).
> I ran Memtest on the problematic server, no problems detected.
> Anyways, I've been sampling the httpd process count, tomcat thread count,
> and free memory (sans cache and buffers) every minute via a cron job and
> writing the results to a file on both machines since this issue arose 
> about
> 2 weeks ago.  My hope is that the next time it bugs out, i'll be able to 
> say
> "oh the thread count exploded, but the memory use only increased
> incrementally" or whatever the cause actually is.
> Have you done any monitoring at that fine grained a level on your systems?
> It would be especially interesting to see how your old HW setup compares 
> to
> the new ones.
> Hope I've helped a little bit :)
> On Fri, Feb 5, 2010 at 11:26 AM, Carl <> wrote:
>> Chris,
>> I agree but this seemed like a low risk change.  I also understand that
>> changing too many things at once sometimes confuses the issue.
>> Thanks,
>> Carl
>> ----- Original Message ----- From: "Christopher Schultz" <
>> To: "Tomcat Users List" <>
>> Sent: Friday, February 05, 2010 10:48 AM
>> Subject: Re: Tomcat dies suddenly
>>> Hash: SHA1
>>> Carl,
>>> On 2/5/2010 10:16 AM, Carl wrote:
>>>> Do you see any harm in just doubling the number (to 2048) just to see 
>>>> if
>>>> it has an impact?
>>> We would expect to see an OOME (strange, I know) if you were running out
>>> of file descriptors.
>>> - -chris
>>> Version: GnuPG v1.4.10 (MingW32)
>>> Comment: Using GnuPG with Mozilla -
>>> AMkAn1H2romyi8eFunc3lF+eDiQa0Hlq
>>> =muVu
>>> -----END PGP SIGNATURE-----
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message