Problem Statement

Memory leak in web application running on Tomcat

 

System Information

Tomcat 5.5.27

Apache HTTPD 2.2.9

Java HotSpot(TM) Server VM (build 1.5.0_14-b03, mixed mode)

Mod_proxy_http connector

Solaris 10 x86

Tomcat JVM heap settings: min:2GB and max:2GB

 

Observations and Investigation Done

Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern.

 

Heap dumps were gathered to determine the root cause and observations are as below from the dumps:

  1. HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space.

org/apache/jasper/compiler/JspRuntimeContext

Java/util/Collections$SynchronizedMap

Java/util/HashMap

Java/util/HashMap$Entry

  1. Heap dumps gathered during different times and after a Full GC always indicate 100 entries
  2. The number of objects referenced by each HashMap entry vary between 2-3
    1. These are either a String and JspServletWrapper object

or

    1. A String, JspServletWrapper and another HashMap Entry. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes

The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below

 

 

  1. The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array

 

Questions

  1. I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause?
  2. I google and found the following interesting links
    1. http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html Is there any know issues, configuration that causes a memory leak by caching of servlet responses in the container and not flushing the cached objects?
    2. http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E We use a lot of JSTL in our web application. Are there any known issues around memory issues as indicated on the link?

 

Any inputs to help identify the root cause of this problem would be highly appreciated.

 

Kind Regards

Anurag

-------------------------------------------------------------------
Anurag Kapur

Senior Associate - Technology,
Sapient Corporation.

http://www.linkedin.com/in/anuragkapur

http://www.google.com/profiles/anuragkapur
-------------------------------------------------------------------