river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: ClassLoader and Class Garbage Collection issues with Serialization.
Date Wed, 14 Apr 2010 02:45:10 GMT
Awesome, thank you very much Peter, you've saved me countless hours.  & 
Thanks for the test example.

So in reality if I implement a stream based lookup implementation that 
returns results, ordered by their packages versions, such that 
Classloaders are shared for common code 1.  Which would control 
ClassLoading of different proxies, such that those with commonality are 
dealt with first and discarded by the client, leaving garbage collection 
to clean up, clients might run out of memory?  The goal is to enable 
lookups to be performed over the internet proper, enabling massive 
result sets to be returned as a stream for inspection.

Could DGC be relied upon for Remote Method Invocation to prevent used 
classes from being garbage collected, if ObjectStreamClass was utilising 
weak references?

Serialization alone, without Remote Method Invocation and DGC, would 
have this problem of class files and their compute expensive descriptors 
becoming garbage collected too often with weak references.

In my case, certainly on the CDC platform if I utilise standard Java 
Serialization, the client risks running out of memory? 

Are they any jvm options that can be used to increase the memory reserve 
that triggers garbage collection of soft referents?



1.  This has implications for trust, since typically Permission is 
assigned to a PermissionDomain -> ClassLoader -> Codebase, if several 
services shared the same Codebase, I would have to rely on Principals 
and Codebase Signers for trust. 

Trust would rely on:

    * Permissions Granted based on codebase signer (Upper bound of trust
      for the client).
    * ProxyTrustVerifier - does the Service trust the proxy (Is this the
      Services proxy instance)?
    * Principal: the Client can authenticate itself with the Service node.
    * Principal: the Service node can authenticate itself with the
      client (How can a services Permissions be restricted based on
      identity when several services share the same codebase at the client)?

Peter Jones wrote:
> [Everything I say below is about Sun's JDK/JRE implementation, although I believe that
IBM's inherits the same serialization code.]
> Since the fix for 5056445/6232010, in JDK 6 and 5.0u7, the static class descriptor cache
in java.io.ObjectStreamClass does not maintain strong references to serializable classes that
have been used.  A potential confusion, though, is that it does maintain "soft" reachability
to them, which is stronger than "weak" (see the java.lang.ref package doc), and which means
that such classes (and thus their defining class loaders) might only be garbage collected
if there is sufficient pressure on the JVM heap that their storage is needed to satisfy other
allocations.  In other words, the collector merely noticing that the classes are not strongly
reachable would not not sufficient for them to be collected. 
> See below an edit of that SerializationTest.java, updated to use a reference queue to
monitor the garbage collection of the class in a non-IBM-specific way.  It should pass, with
JDK 6 and 5.0u7, but note the invocation of flushSoftRefs necessary to force sufficient heap
pressure to cause soft references to be cleared-- System.gc alone is not sufficient.
> Some background: The ObjectStreamClass implementation faces a dilemma about whether to
maintain soft or weak references to the cached class descriptors (which in turn reference
their corresponding classes).  It could use weak references instead, but then cache entries
would likely be cleared very soon after a given serialization activity has completed, going
against the purpose of caching these expensive-to-compute descriptors.  What it would really
like to do is make its referencing of a class descriptor conditional on the reachability of
the corresponding class itself, but that is not doable with the current java.lang.ref API.
  Doing that sort of thing is the subject of this RFE (which is probably my personal favorite
open Java libraries RFE):
> 	http://bugs.sun.com/view_bug.do?bug_id=4630118
> Even before the 5056445/6232010 fix, though, the situation wasn't as bad as suggested
by the article you quoted.  The serializable classes were indeed strongly referenced by the
static cache, as keys, but the corresponding class descriptor values were (again) softly referenced,
and the cache would clear out entries whose soft references had been cleared.  The problem
was that this clearing did not happen in the background, rather it required some cache usage
to occur, so garbage collection of unused serializable classes required ongoing serialization
activity.  Also, this clearing was not in a closed loop with the garbage collector, thus risking
OutOfMemoryError in tight situations.
> -- Peter
> On Apr 10, 2010, at 9:24 PM, Peter Firmstone wrote:
>> I have stumbled across a troubling problem with Serialization relating to Garbage
Collection of Classes and ClassLoaders and was hoping someone might be able to shed some light
on the issue.
>> Is it really true that the more objects you distribute, the greater your memory consumption
because Class files and ClassLoaders cannot be garbage collected?
>> Regards,
>> Peter.
>> The issue can be found here:
>> http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
>> And here's the relevant information, pasted from the link:
>> Problems related to garbage collection and serialization
>> The garbage collector interacts closely with the class loader. Among other things,
the collector examines the class loader data structures to determine which classes are /live/
-- that is, are not garbage collectable. This can often lead to some unexpected problems.
>> Figure 2 illustrates a situation where serialization affects the garbage collection
(GC) of classes and a class loader in an unexpected way:
>>   *Figure 2. Serialization example*
>> Serialization example
>> In this example, |SerializationTest| instantiates a |URLClassLoader|, called |loader|.
After loading |SerializationClass|, the class loader is dereferenced. The expectation is that
this will allow the classes loaded by it to be garbage collected. The code for these classes
is illustrated in Listings 9 and 10:
>> *Listing 9. SerializationTest.java*
>> import java.net.MalformedURLException;
>> import java.net.URL;
>> import java.net.URLClassLoader;
>> public class SerializationTest extends ClassLoader {
>>  public static void main(String args[]) {
>>     try {
>>        URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
>>              "file://C:/CL_Article/Serialization/dir1/") });
>>        System.out.println("Loading SerializationClass");
>>        Class c = loader.loadClass("SerializationClass");
>>        System.out.println("Creating an instance of SerializationClass");
>>        c.newInstance();
>>        System.out.println("Dereferencing the class loader");
>>        c = null;
>>        loader = null;
>>                System.out.println("Running GC...");
>>        System.gc();
>>        System.out.println("Triggering a Javadump");
>>        com.ibm.jvm.Dump.JavaDump();
>>             } catch (MalformedURLException e) {
>>        e.printStackTrace();
>>     } catch (InstantiationException e) {
>>        e.printStackTrace();
>>     } catch (IllegalAccessException e) {
>>        e.printStackTrace();
>>     } catch (ClassNotFoundException e) {
>>        e.printStackTrace();
>>     }
>>  }
>> }
>> *Listing 10. SerializationClass.java*
>> import java.io.File;
>> import java.io.FileOutputStream;
>> import java.io.ObjectOutputStream;
>> import java.io.Serializable;
>> public class SerializationClass implements Serializable {
>>   private static final long serialVersionUID = 5024741671582526226L;
>>   public SerializationClass() {
>>       try {
>>           File file = new File("C:/CL_Article/Serialization/test.txt");
>>           FileOutputStream fos = new FileOutputStream(file);
>>           ObjectOutputStream oos = new ObjectOutputStream(fos);
>>           oos.writeObject(this);
>>           oos.reset();
>>           oos.close();
>>           fos.close();
>>           oos = null;
>>           fos = null;
>>           file = null;
>>       } catch (Exception e) {
>>           e.printStackTrace();
>>       }
>>   }
>> }
>> Using a Javadump, it is possible to discover whether the class loader has been garbage
collected. (See the first article in this series for more on using Javadump.) If the following
section appears in the list of class loaders, then it has not been collected:
>> ------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
>>       Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0)        Number of loaded
classes 1        Number of cached classes 11             Allocation used for loaded classes
1             Package owner 0x00ADB6D8
>> Though dereferencing a user-defined class loader seems like a way to ensure that
the classes are garbage collected, this is not actually the case. In the previous example,
the problem stems from the use of |java.io.ObjectOutputStream.writeObject(Object obj)| and
its implications on GC.
>> When |writeObject()| is invoked (to serialize |SerializationClass|), a reference
to this class object is passed internally to |ObjectStreamClass| and stored in a lookup table
(that is, in an internal cache). This reference is kept to speed up future serialization of
the same class.
>> When the class loader is dereferenced, the classes that it loaded are not garbage
collectable. This is because there is a live reference to the |SerializationClass| class from
the |ObjectStreamClass| lookup table. |ObjectStreamClass| is a primordial class and therefore
is never garbage collected. The lookup table is referenced from a static field in |ObjectStreamClass|
and is kept in the class itself rather than in an instance of it. As a result, the reference
to |SerializationClass| exists for the lifetime of the JVM, and the class thus cannot be garbage
collected. Importantly, the |SerializationClass| class has a reference to its defining class
loader, and so it cannot be completely dereferenced either.
>> To avoid this problem, any classes that are to be serialized should be loaded by
a class loader that does not need to be garbage collected -- by the system class loader, for
> import java.lang.ref.Reference;
> import java.lang.ref.ReferenceQueue;
> import java.lang.ref.WeakReference;
> import java.net.MalformedURLException;
> import java.net.URL;
> import java.net.URLClassLoader;
> import java.util.ArrayList;
> import java.util.List;
> public class SerializationTest extends ClassLoader {
>     public static void main(String args[]) throws Exception {
>         URLClassLoader loader = new URLClassLoader(new URL[] {
>             new URL("file:dir1/")
>         });
>         System.out.println("Loading SerializationClass");
>         Class c = loader.loadClass("SerializationClass");
>         System.out.println("Creating an instance of SerializationClass");
>         c.newInstance();
>         ReferenceQueue<Class<?>> queue = new ReferenceQueue<Class<?>>();
>         Reference<Class<?>> ref = new WeakReference<Class<?>>(c,
>         System.out.println("Dereferencing the class loader");
>         c = null;
>         loader = null;
>         System.out.println("Running GC...");
>         System.gc();
>         //System.out.println("Triggering a Javadump");
>         //com.ibm.jvm.Dump.JavaDump();
>         flushSoftRefs();
>         Reference<? extends Class<?>> dequeued = queue.remove(1000);
>         if (dequeued == ref) {
>             System.out.println("SerializationClass garbage collected");
>         } else {
>             throw new Error("SerializationClass not garbage collected");
>         }
>     }
>     private static void flushSoftRefs() {
>         try {
>             List<Object> l = new ArrayList<Object>();
>             while (true) l.add(new byte[10000]);
>         } catch (OutOfMemoryError e) {
>             System.out.println("memory exhausted");
>         }
>     }
> }

View raw message