Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: domain of rainer.jung@kippdata.de
 designates 195.227.30.149 as permitted sender)
Message-ID: <4ACB315B.8060509@kippdata.de>
Date: Tue, 06 Oct 2009 14:00:27 +0200
From: Rainer Jung <rainer.jung@kippdata.de>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de;
 rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4
MIME-Version: 1.0
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: java.lang.OutOfMemoryError: Java heap space
References: <2185441d0910031107y7d98d4belc1ff51f2e1d5a23a@mail.gmail.com>
 <4AC796ED.4000009@kippdata.de>
 <2185441d0910041245g4cc65714y307585e55a57e93d@mail.gmail.com>
	 <4AC90736.5040004@kippdata.de>
 <2185441d0910050958h22a99400tfcd9aab81ba4ce4a@mail.gmail.com>
In-Reply-To: <2185441d0910050958h22a99400tfcd9aab81ba4ce4a@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 7bit

On 05.10.2009 18:58, Joe Hansen wrote:
> Thank you so much for your tips, Rainer!
> 
> The websites went down yet again. Increasing the java heap size took
> care of the OutOfMemoryError, but the number of httpd processes keep
> increasing until the websites crash. I haven't added any new code in
> the past few months, hence I am surprised why the requests are getting
> stuck. Here's a link to the tomcat thread dumps:
> http://pastebin.com/m17eea139

Just tried to look at it, but pastebin replies with:

====================
Down for maintenance - 6th Oct 2009

Pastebin.com is getting an unprecedented amount of traffic due to a news
story in which some leaked Hotmail passwords have been pasted on this site

...
====================

Let's see, when they will be up again. They're running Apache 1.3.33 ...

> Please let me know if you cannot view it and I will email the relevant
> portion of the catalina.out file to you. Is there an easy way to find
> out what code is causing the requests to get stuck?
> 
> Thank you!
> 
> Joe
> 
> 
> On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung <rainer.jung@kippdata.de> wrote:
>> Hi Joe,
>>
>> On 04.10.2009 21:45, Joe Hansen wrote:
>>> Rainer, Thank you so much for your kind reply!
>>>
>>> I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
>>> hoping that would fix the issue. I had configured our webserver to use
>>> Jasig's Central Authentication System (CAS). Recently I increased the
>>> session timeout from 30 minutes to 4 hours. I am guessing that must
>>> have had an impact on the number of tickets that the CAS could store
>>> in the Java's memory space.
>>>
>>> I did run the kill -QUIT command against the tomcat process. It did
>>> generate a huge output in the catalina.out file. I am unable to
>>> decipher it. I do not want to post it to the mailing list because its
>>> very long. Would you be able to please tell me what should I be
>>> looking for within this long thread dump?
>>
>> Can you put it somewhere on the web, so we can look at it, or are you
>> afraid there is something private in there? You could use pastebin or
>> something similar in case you do not have a public web server yourself.
>>
>> If you don't want to post in public, you can also mail it to me, I will
>> post the result, in case I find something relevant.
>>
>> Regards,
>>
>> Rainer
>>
>>> On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung <rainer.jung@kippdata.de> wrote:
>>>> On 03.10.2009 20:07, Joe Hansen wrote:
>>>>> Hey All,
>>>>>
>>>>> I get this error (java.lang.OutOfMemoryError: Java heap space) after
>>>>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
>>>>> for a few hours. This problem started just since two days. Never had
>>>>> this issue before!
>>>>>
>>>>> I have also noticed that as soon as I startup the server, 9 httpd
>>>>> processes start. Number of httpd processes keep on increasing until I
>>>>> get the OutOfMemoryError.
>>>>> $ps -aef | grep httpd
>>>>> root     31984     1  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31987 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31988 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31989 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31990 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31991 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31992 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31993 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>> apache   31994 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
>>>>
>>>> Sounds like requests get stuck or responses are only returned very slowly.
>>>>
>>>> I would take thread dumps during the time requests pile up (e.g. httpd
>>>> process count increases). Thread dumps are generated by "kil -QUIT"
>>>> against the Tomcat process. Result is written to catalina.out. Always
>>>> take afew thread dumps shortly after each other, e.g. 3 dumps each 3
>>>> seconds apart from the previous one, so that you can find out, if a
>>>> status in a dump is pure coincidence or lasts for somewhat longer.
>>>>
>>>>> $ps -aef | grep tomcat
>>>>> root     31949     1 43 11:23 pts/0    00:00:58 /usr/java/jdk/bin/java -Djava.u
>>>>> il.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.co
>>>>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties -Djava.endorsed.dirs=/u
>>>>> r/lib/apache-tomcat/common/endorsed -classpath :/usr/lib/apache-tomcat/bin/boot
>>>>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/us
>>>>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat -Djava.io.tmpdir=/usr
>>>>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>>>>
>>>> There is no Java memory configuration included above (i.e. al defaults).
>>>> It might well be, that you have to explicitely set heap size, perm size
>>>> and if you like also eden and semi spaces.
>>>>
>>>>> Can someone on this list please help me resolve this issue.
>>>>>
>>>>> Thanks you,
>>>>> Joe
>>>>
>>>> Regards,
>>>>
>>>> Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org