tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl" <c...@etrak-plus.com>
Subject Re: Tomcat dies suddenly
Date Sat, 06 Feb 2010 00:45:38 GMT
Jonathon,

I have yet to do any sampling, I have been visually monitoring the servers 
with the VisualJVM, the Slackware system monitor, top and vmstat.  I was 
using JMeter to stress test the application.

The old server is Slackware 12.1 (32 bit), Tomcat 5.5, Java 1.5.0_01.

The new servers are Slackware 13 (64 bit), Tomcat 6.0.24, Java 1.6_18.

The old server has run flawlessly for several years.  The new servers are 
kicking my butt:

- They will run for a period of time (could be 10 days, could be 15 minutes) 
and silently fail, no error logs, no memory dump, nothing in catalina.out, 
no tracks that I have been able to find.

- The issue seems load related and cumulative.  A week ago was a very heavy 
processing day and the server failed repeatedly starting about noon. 
Otherwise, it would usually fail late in the day.

-  I have tried different heap sizes.  My first inclination was a larger 
heap but, as Chuck pointed out, that was probably the wrong thing.  I am 
currently running with a 500MB heap (384MB permGen.)  Been up for two days 
and looks great.

-  I have quite a bit of logging to catalina.out because I was, at one 
point, looking for a connection leak.  I also started incremental GC and 
printing the GC instances using the appropriate java flags.  The last entry 
for the last failure was a GC.

-  I tried provoking the failure by stressing the application using 
JMeter... wouldn't fail even when I forced an OOM.

Your idea of sampling is interesting... may try that next.

Thanks,

Carl

----- Original Message ----- 
From: "Jonathan Mast" <jhmast.developer@gmail.com>
To: "Tomcat Users List" <users@tomcat.apache.org>
Sent: Friday, February 05, 2010 7:11 PM
Subject: Re: Tomcat dies suddenly


> Hi Carl, I've skimmed thru all your posts, please excuse me if these
> questions are redundant.
>
> You've mentioned VisualJVM, JMeter, and Slackware monitoring tools, it 
> seems
> to me that you are using these to occasionally to monitor you Tomcat
> instance, when things seem to be going badly (thats my perception from
> reading your posts)
>
> Have you done any long term sampling (Tomcat threads, memory, etc) of your
> system?
>
> I use Slackware 12.2 (32-bit), Httpd 2.2, Java 1.6, Tomcat 6.0.18 and had
> Tomcat die in a manner similiar to yours:  First it died silently (nothing
> in the catalina.out), the next time (about a week later) it became
> non-responsive complaining about PermGen, OOMException and so on.  I 
> haven't
> determined the cause yet, I rolled over to another server (exact same
> hardware/software btw).
>
> I ran Memtest on the problematic server, no problems detected.
>
> Anyways, I've been sampling the httpd process count, tomcat thread count,
> and free memory (sans cache and buffers) every minute via a cron job and
> writing the results to a file on both machines since this issue arose 
> about
> 2 weeks ago.  My hope is that the next time it bugs out, i'll be able to 
> say
> "oh the thread count exploded, but the memory use only increased
> incrementally" or whatever the cause actually is.
>
> Have you done any monitoring at that fine grained a level on your systems?
>
> It would be especially interesting to see how your old HW setup compares 
> to
> the new ones.
>
> Hope I've helped a little bit :)
>
> On Fri, Feb 5, 2010 at 11:26 AM, Carl <carl@etrak-plus.com> wrote:
>
>> Chris,
>>
>> I agree but this seemed like a low risk change.  I also understand that
>> changing too many things at once sometimes confuses the issue.
>>
>> Thanks,
>>
>> Carl
>>
>> ----- Original Message ----- From: "Christopher Schultz" <
>> chris@christopherschultz.net>
>> To: "Tomcat Users List" <users@tomcat.apache.org>
>> Sent: Friday, February 05, 2010 10:48 AM
>>
>> Subject: Re: Tomcat dies suddenly
>>
>>
>>  -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Carl,
>>>
>>> On 2/5/2010 10:16 AM, Carl wrote:
>>>
>>>> Do you see any harm in just doubling the number (to 2048) just to see 
>>>> if
>>>> it has an impact?
>>>>
>>>
>>> We would expect to see an OOME (strange, I know) if you were running out
>>> of file descriptors.
>>>
>>> - -chris
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.10 (MingW32)
>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>>
>>> iEYEARECAAYFAktsPbYACgkQ9CaO5/Lv0PBWnwCfXyZVJNsd/mMCgJWVamRBVZ+F
>>> AMkAn1H2romyi8eFunc3lF+eDiQa0Hlq
>>> =muVu
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message