tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Date Tue, 02 Apr 2013 19:58:53 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Saumil,

On 4/1/13 11:01 PM, saumil shah wrote:
> I recently deployed one of the COTS products SAP Business Objects. 
> When the product was deployed , everything seemed to run fine but 
> yesterday we started experiencing "Service is unavailable" error , 
> upon enabling DEBUG logs in Tomcat , we saw the error below with 
> memory leak message. We were wondering what could have caused it 
> ....
> 
> The Tomcat is 64 bit , JVM is 64 bit but the applications deployed 
> are 32 bit webapps .... we say Tomcat6 process becoming
> unresponsive around 2GB mark.
> 
> 
> Apr 1, 2013 8:53:18 PM org.apache.coyote.http11.Http11Protocol 
> pauseINFO: Pausing Coyote HTTP/1.1 on http-8080Apr 1, 2013 8:53:19
> PM org.apache.catalina.core.StandardService stopINFO: Stopping
> service CatalinaApr 1, 2013 8:53:19 PM 
> org.apache.catalina.loader.WebappClassLoader 
> clearReferencesThreadsSEVERE: The web application 
> [/AnalyticalReporting] appears to have started a thread named 
> [Business Objects - Sessions Clean up] but has failed to stop it. 
> This is very likely to create a memory leak.

This means what it says it means: your webapp (not Tomcat) started a
thread that wasn't stopped by the time the webapp was stopped. This
will pin the webapp's ClassLoader in memory and you'll get a whole
bunch of stuff that can't be GC'd. The simple explanation is that you
need to be sure that your threads are stopped. The solution may be
complicated.

Presumably any component that starts a thread also knows how to stop
that same thread. Check the documentation for those components to make
sure that you are shutting them down properly (usually in a
ServletContextListener, etc.).

Sometimes, it takes a few seconds for a thread to complete its work
and actually terminate, and the messages you get from Tomcat are
inaccurate. That's for you to determine.

> Apr 1, 2013 8:53:19 PM 
> org.apache.catalina.loader.WebappClassLoader 
> clearReferencesThreadsSEVERE: The web application 
> [/AnalyticalReporting] appears to have started a thread named 
> [AWT-Windows] but has failed to stop it. This is very likely to 
> create a memory leak.

Read the documentation for the JreMemoryLeakLeakPreventionListener.
This is a standard Tomcat component that can be used to work-around
well-known memory leaks, etc. that are triggered by certain JVM calls.
There are specific configurations for preventing AWT threads from
being problematic, but they are not enabled by default. You probably
want to enable them if you are using AWT for anything.

> Apr 1, 2013 8:53:19 PM 
> org.apache.catalina.loader.WebappClassLoader 
> clearReferencesThreadsSEVERE: The web application 
> [/AnalyticalReporting] appears to have started a thread named 
> [ORBacus:Client:SenderThread] but has failed to stop it. This is
> very likely to create a memory leak.

Anything with "ORB" in the name usually means a CORBA-related
component: check to see that anything that start threads is also
stopping them.

> Apr 1, 2013 8:53:19 PM 
> org.apache.catalina.loader.WebappClassLoader 
> clearThreadLocalMapSEVERE: The web application
> [/AnalyticalReporting] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd])
> and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser]
> (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@228de67]) but
> failed to remove it when the web application was stopped. This is
> very likely to create a memory leak.

Looks like your Business Objects version has a bug in it that does not
clear-out ThreadLocal values when requests are completed. Talk to them
about this problem.

> Apr 1, 2013 8:53:20 PM 
> org.apache.catalina.loader.WebappClassLoader 
> clearThreadLocalMapSEVERE: The web application [/VoyagerClient] 
> created a ThreadLocal with key of type 
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl]
>
> 
(value
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@13123b1c])
>
> 
and a value of type
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter]
>
> 
(value
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@11e6b50c])
>
> 
but failed to remove it when the web application was stopped. This is
> very likely to create a memory leak.Apr 1, 2013 8:53:20 PM 
> org.apache.catalina.loader.WebappClassLoader 
> clearThreadLocalMapSEVERE: The web application [/VoyagerClient] 
> created a ThreadLocal with key of type 
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl]
>
> 
(value
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@41783002])
>
> 
and a value of type
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter]
>
> 
(value
> [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@3997f4e2])
>
> 
but failed to remove it when the web application was stopped. This is
> very likely to create a memory leak.

Looks like AspectJ has a similar problem: ThreadLocals need to be
removed from threads after every request.

You are probably seeing a pattern, here, so I won't bother looking at
every other error message you have posted.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRWzh6AAoJEBzwKT+lPKRYUoQQALSx4IVs9Pd/QfAQKeB6klIP
9bfoG8bRk/fcqdQiJfaNJGw8xaKm0WVMalYB/U1xH7dZSdFDlz4LoQuitRLEMnsX
WyRspvI3N3KUIhGUuU9GiVz0nn1vwaQF87K81g/7IpCOKcFsWfruHnict2UMjrg1
pEI7xiNoC3oPsbd6+APhAy9JUip8gt4j7I0VUVAvCOg4L7PLvKGY3y4TXy594uFO
D/QZ6cN/9SdImOO9wunwHk1+Ot3B3ZKuurdbJk6djXyt6K2Et4lHGJ8KldhZfgfj
QdQjRg/ybTune+gWJ9QTPo5p7FOgTJxS2x94a+pXx9aTkCBqbBiKCWWU47O+wtf4
P5Nc71m0N1zeLsSjThjDoaH8DA77AbwtNWHQEwemvqp6y8CQy8ypp/f58lrzdOza
lFhER7mMtzlNNCHdxZKUEY9yBaEIzWAV9vPP51yWEUBCJaGRZ9icTZNvIQPUAcLB
JzoIfgEHFjrVlLXCKb0phIq6He0B/yDZ4xwA2rn9bXKhRJEuZbBoYXTM6jsRJDNk
ab/wMZKXfK5On5N9U9TVj3kzGni35GI2wApJ9OGvYWdKv+9oX4vKQNOl8xAdm8IM
wAE3ooZk0XvsotLphMP+V9PHXNCaeZeWORsNKKUlkxZDAPU5vdmaVfcdU0um2lm2
yqAz4cJjThxGwbh62xe3
=UtMt
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message