tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Mark_Desp...@McAfee.com>
Subject RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Date Wed, 22 Apr 2009 02:27:22 GMT
Ok, so my wife actually wrote a couple of month ago in Japanese about using strategy for leveraging
the Insane library and a continuous integration server in order to prevent webapp classloader
leakage issues from creeping in.  If you can read Japanese, check out
  http://pdxwhitebox.blog118.fc2.com/blog-entry-33.html

For those of us who can't read Japanese, today she posted the English text it was based upon
here:
  http://pdxwhitebox.blog118.fc2.com/blog-entry-161.html

The information there is a little light on the specifics, but it does a great job at describing
the strategy, the rationale, where to find Insane, and which API to use.  When you read it,
treat "plugins" as interchangeable with "webapps".

Getting a little more specific, we leveraged Insane's ScannerUtils.scan() method to search
for webapp classloaders that should not be reachable via a strong reference.  Doing this involved
(1) implementing a org.netbeans.insane.scanner.Visitor that gathers relevant classloaders
and (2) implementing a org.netbeans.insane.scanner.Filter that causes weak/soft/phantom references
to be skipped over by ScannerUtils.scan().  

(1) To implement the Visitor class, just have visitObject(ObjectMap map, Object object) collect
relevant classloaders passed as the second argument (probably instances of org.apache.catalina.loader.WebappClassLoader).
 All other methods can have empty implementations.

(2) To implement the Filter class, have accept(Object obj, Object referredFrom, Field reference)
return false if the second argument is an instance of SoftReference, WeakReference, or PhantomReference.
 Return true for everything else.


With the Visitor and Filter implemented, the scanning code would look something like the following:


        Set<Object> roots = ScannerUtils.interestingRoots();
	  // add to this set anything else you'd like to make sure gets visits

        MyWebAppClassLoaderGatherer visitor = new MyWebAppClassLoaderGatherer();
        MySkipWeakReferencesFilter filter = new MySkipWeakReferencesFilter();
        ScannerUtils.scan( filter, visitor, roots, true );

	  // do whatever is needed with the classloaders gathered by the visitor

Unfortunately, I don't remember why the last argument to ScannerUtils.scan() should be true,
and I don't have the Insane source or javadoc handy to remind me.  Sorry about that...

With this in place, you can then setup your test environment to exercise a given webapp, shut
it down, and then invoke your ScannerUtils code to see if that the webapp's classloader is
still hanging around.  This seems to work well with in combination with a continuous integration
(CI) server whose job is to run tests against checkins all day.  If your CI server tracks
who checked it what, you'll be able to quickly narrow down what may have caused the leak,
assuming that your web app wasn't leaking a classloader to begin with.
 
A word of warning... this is a very heavy weight operation.  If there are better ways this
can be done, I'd love to hear about it!

Best regards, 

Mark DeSpain
 

-----Original Message-----
From: Despain, Mark 
Sent: Tuesday, April 21, 2009 4:01 PM
To: users@tomcat.apache.org
Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"

Hi Chris,

I'll follow up later tonight.  Hopefully I'll have less typos then, but don't be surprised
if I just go triple-negative instead :)

~Mark
 

-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net] 
Sent: Tuesday, April 21, 2009 3:47 PM
To: Tomcat Users List
Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Mark,

On 4/21/2009 6:17 PM, Mark_Despain@McAfee.com wrote:
> None of the issues I've looked into have never been attributed to
> Tomcat.

You mean "ever" attributed to Tomcat, right? Good. ;)

> * A webapp registering an object with another object that outlives
> the webapp and forgetting to unregister it webapp shutdown.  As a
> result, that object cannot be garbage collected, which also prevents
> that webapp's classloader from being garbage collected.

It's worse than that: even if most of the objects created from classes
loaded by that ClassLoader have gone out of scope and can be GC'd, the
java.lang.Class, java.lang.Package, and java.lang.reflect.* objects that
are created from those loaded classes won't be released until the
ClassLoader is released. So if you have a single object that is being
held by an object loaded by another ClassLoader, you can have
potentially hundreds or thousands of objects left around in memory just
wasting space.

> * Instantiating a new Thread somehow (e.g. directly via its
> constructor, or indirectly via a thread pool, a Timer, or a
> ScheduledExecutorService) in response to a request to a webapp.  This
> is because, by default, the new Thread inherits the parent thread's
> context classloader, which in Tomcat (5.5.x) is set to that webapp's
> classloader.

This is a very good point to consider, because many folks aren't aware
of the thread->classloader relationship. If you don't kill your threads,
you will keep your whole app's ClassLoader around forever. The
'daemonness' of the thread is not relevant. I'm looking at you, Quartz!

> To help pro-actively detect webapp classloader garbage collection
> issues, I've leveraged the Insane library (a library I found from
> Netbeans while researching the topic) to write a utility that
> searches for webapp classloaders that should have been garbage
> collected.  Using the utility in combination with automated tests has
> been definitely helped catch and diagnose issues.

Could you post some more info on this? I'm sure a lot of folks would be
interested in this kind of thing.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAknuTNYACgkQ9CaO5/Lv0PCN6wCgkDKwhzpRB7re4StuClVe1Rt/
3K0Anj5eXjLiTql97dxbhrNFavPXGIvC
=Iuos
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Mime
View raw message