tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirill Kireyev <kir...@instagrok.com>
Subject Re: high CPU usage on tomcat 7
Date Sat, 06 Oct 2012 18:26:15 GMT
Thanks for all your thorough advice Shanti! (and everyone else)

Here are my findings so far:
0) My servlet sessions store a large number (~10s of M) or data in RAM. 
This is by design, to optimize performance. I can also have ~3K active 
sessions at any one time. Hence a large heap size.
1) When I (1) manually expire inactive sessions through Tomcat web 
interface and (2) manually hit "Perform GC" through jvisualvm console 
attached to the tomcat process, everything works great, and the memory 
is successfully reclaimed, and the used heap  size drops back to what it 
should be when the application initializes.
2) However, this doesn't seem to work automatically. More specifically:
     a) Sessions are not expiring without manually doing it (I can see 
the number growing in the Tomcat web interface). Even though my 
conf/web.xml says:
     <session-config>
         <session-timeout>20</session-timeout>
     </session-config>
     b) A full garbage collection is not being performed unless I do it 
manually. I'm attaching the GC records from my logs/catalina.out.

Any insights?

Thanks a lot!
Kirill


On 10/5/12 7:40 AM, Shanti Suresh wrote:
> Hi Kirill,
>
> Do you happen to have a QA or a test environment by any chance?
>
> 25GB is very big for a JVM heap size.
>
> Some questions I have are:
> (1) What does the application do?   How many concurrent users do you have?
> (2) What was the heap set at before you increased it to 25GB?
> (3) Why did you think you needed 25GB?  Is this just an arbitrary size
> because your machine has 32GB RAM total and you wanted to allocate out the
> JVM size to maximum available?
> (4) What symptoms did you face before you increased the heap to 25 GB?
> (5) What size are the heapdumps that the earlier script generated?  It
> would be a challenge to troubleshoot a heapdump of 20 - 25 GB.
>
> Suggestions for diagnostic steps:
> (1) The first step, I think, is to look at the JConsole memory use graph
> after application restart and see if the memory is rising linearly
> throughout the application's use.  You seem to have a memory leak like
> Charles pointed out.
> (2) Restart QA.  Bring up JConsole.  Have only one user use the system.
> Study JConsole Overview panel - Memory, CPU, threads, #loaded classes.
> (3) Take a heapdump.
> (4) Have two users use QA.  Observe graphs.  Take heapdump.
> (5) Have three users use QA.  Observe graphs.  Take heapdump.
> (6) Open each heapdump using Eclipse Memory Analyzer.  MAT has useful
> features.  One of them is to generate a leak suspect report.  By just
> viewing the available graphs, you'll get an idea of what classes are
> increasing in size and not being reclaimed.
>
> Others may have other useful tips.
>
> Thanks.
>
>                             -Shanti
>
> On Thu, Oct 4, 2012 at 2:08 PM, Jeff MAURY <jeffmaury@gmail.com> wrote:
>
>> Le 4 oct. 2012 14:38, "Caldarale, Charles R" <Chuck.Caldarale@unisys.com>
>> a
>> écrit :
>>>> From: Kirill Kireyev [mailto:kirill@instagrok.com]
>>>> Subject: Re: high CPU usage on tomcat 7
>>>> Perhaps what I need is to have the JVM do garbage collections more
>>>> frequently, so that they don't become a huge CPU-hogging ordeals
>>>> when they do happen.
>>> That's not how it works.  The amount of time spent in a GC operation is
>> dependent almost entirely on the number of live objects in the heap, not
>> the amount of space being reclaimed.  Having a smaller heap might improve
>> your cache hit rate, but it will have little effect on the time used by GC.
>>> It is highly likely that your webapps have a memory leak.  You need to
>> use a profiler and find out what's eating up all that memory.  If you set
>> the heap size to a smaller value, you should be able to provoke the
>> situation earlier, but it certainly won't fix it.
>> +1
>>
>> Jeff
>>>   - Chuck
>>>
>>>
>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
>> MATERIAL and is thus for use only by the intended recipient. If you
>> received this in error, please contact the sender and delete the e-mail and
>> its attachments from all computers.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>


-- 
*Kirill Kireyev, PhD*
Founder/CTO instaGrok.com <http://www.instagrok.com>
kirill@instagrok.com
Twitter: @instaGrok <http://twitter.com/InstaGrok>
FB: facebook.com/instagrok <http://www.facebook.com/instagrok>
<http://www.instagrok.com>

Mime
View raw message