tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Hansen <joe.hansen...@gmail.com>
Subject Re: java.lang.OutOfMemoryError: Java heap space
Date Tue, 06 Oct 2009 17:44:33 GMT
Rainer,

I spoke to soon! As I suspected, the problem isn't fixed yet and the
websites crashed again. This time I took three thread dumps at the
time tomcat was down. Here they are: http://pastebin.com/m2a7e1198

I will learn from your previous analysis of the thread dumps and I try
to understand what's happening.

Thanks,
Joe

On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen <joe.hansen.at@gmail.com> wrote:
> Rainer,
>
> Thanks for looking at those long thread dumps for me!!
>
> I am sorry. I did NOT take these dumps at the right time (i.e. when
> Tomcat was inundated with requests and couldn't cope with the load).
> After I increased the heap size to 512MB (from 64MB default), I am not
> getting the OutOfMemoryError(s) anymore. After I set KeepAlive On
> (Thanks Andre!), the number httpd processes isn't increasing either.
> The number of httpd processes increased from 8 to 21 and stayed there
> for more than 16 hours now. If the number of httpd processes gets out
> of control again, I will definitely take thread dumps once again.
>
> However, I doubt the issue is fixed because I should have seen this
> issue long time ago (since I haven't changed any code nor the traffic
> to our websites increased in a long while).
>
> Should I just wait and see or are there any tests that I can do?
>
> Your contribution to this forum is amazing, Rainer. I am grateful to
> you and Andre for your efforts. Thank you!
>
> Regards,
> Joe
>
>
>
> On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung <rainer.jung@kippdata.de> wrote:
>> On 05.10.2009 18:58, Joe Hansen wrote:
>>> Thank you so much for your tips, Rainer!
>>>
>>> The websites went down yet again. Increasing the java heap size took
>>> care of the OutOfMemoryError, but the number of httpd processes keep
>>> increasing until the websites crash. I haven't added any new code in
>>> the past few months, hence I am surprised why the requests are getting
>>> stuck. Here's a link to the tomcat thread dumps:
>>> http://pastebin.com/m17eea139
>>>
>>> Please let me know if you cannot view it and I will email the relevant
>>> portion of the catalina.out file to you. Is there an easy way to find
>>> out what code is causing the requests to get stuck?
>>
>> The dump file contains three thread dumps.
>>
>> The things all dumps have in common:
>>
>> - 60 threads for the quartz scheduler, all idle
>> - 13 threads in the AJP connection pool, connected to Apache, but idle
>> waiting for the next request to be send (the same threads in all three
>> dumps)
>> - 6 store plus 6 expiry threads of the EHCache, seems idle
>> - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new
>> connection to come in
>> - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting
>> for work
>> - a couple of other normal threads not directly related to request handling
>>
>> So the time you took the three dumps, this Tomcat was completely idle
>> and did not have a single request to handle.
>>
>> If you are completely sure, you took the dumps while there was a storm
>> of requests and your system couldn't cope the load, something has
>> prevented the requests to ever reach Tomcat.
>>
>> I don't have your Tomcat version at hand at the moment, but for some
>> time very special OutOfMemory errors (could not create native thread)
>> lead to a situation, where Tomcat simply wouldn't accept any new
>> connections. Although you report OutOfMemory errors, I'm not directly
>> suggesting that that is your problem here. There might still be a
>> relation though.
>>
>> Are you sure, that you took the dumps for the right Tomcat at the right
>> time?
>>
>> Regards,
>>
>> Rainer
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message