ant-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lee Breisacher" <LBreisac...@seagullsw.com>
Subject RE: OutOfMemory and bug #8689
Date Thu, 03 Jun 2004 13:14:52 GMT
Thanks for the reply, Stefan.  Hmm, it is a bit more complicated than I
included in my initial email. We are invoking the XJC task which is part
of JAXB from Sun (see
http://java.sun.com/webservices/docs/1.3/jaxb/ant.html). In fact, we are
invoking the XJC task 14 times in our ant build. One of the JAXB
developers analyzed what's happening (see email from Kohsuke Kawaguchi
below) and he discovered that all the classes associated with the XJC
task are loaded on each and every invocation of the XJC task (and never
freed), so after 14 invocations, there were more than 18000 classes
loaded. On Java 1.3, this results in OutOfMemory even when using rather
large values for -Xmx (e.g. set ANT_OPTS=-Xmx1024m -Xss32m). 

So, if you can tell me a different way to invoke the XJC task that
doesn't leak classes, or if there is something the JAXB developers can
do in their implementation/declaration of the XJC task, that would be
great. Otherwise, perhaps someone could look at fixing this
AntClassLoader problem?

Thanks,

Lee
 

-----Original Message-----
From: Kohsuke Kawaguchi [mailto:Kohsuke.Kawaguchi@Sun.COM] 
Sent: Wednesday, April 21, 2004 5:08 PM
To: users@jaxb.dev.java.net
Subject: investigated --- Re: xjc 20040405 uses 1.4 API


I was able to reproduce the problem. Indeed it happens sometime with JDK
1.3.

What's strange is that right before OutOfMemoryError, the memory usage
is just around 7MB and nowhere near the VM limit. So initially I thought
maybe someone is doing something stupid like:

    new byte[x];

where x is incredibly large by mistake.

Then I noticed that if I use Windows task manager to monitor the memory
usage, it grows constantly up to around 54MB before it crashes. This
doesn't agree with what -verbose:gc is showing.

So I then used JVMStat
(http://developers.sun.com/dev/coolstuff/jvmstat/),
unfortunately this one only works with JDK 1.4, but even with JDK 1.4, I
found that it does seem like leaking memory (attached.) At this point I
decided to work with JDK 1.4 exclusively.

Next I used -Xrunhprof to get the basic profilng. It actually generated
a huge file, so I only shows the top #20 memory use:

SITES BEGIN (ordered by live bytes) Wed Apr 21 16:24:50 2004
          percent         live       alloc'ed  stack class
 rank   self  accum    bytes objs   bytes objs trace name
    1 13.89% 13.89%  2260072 16685 2260072 16685  3236 [B
    2  7.37% 21.25%  1198856 23189 1198856 23189  3236 [S
    3  5.87% 27.12%   955824 24316  986600 25233  3236 java.lang.Object
    4  5.74% 32.87%   934360 16685  934360 16685  3236 java.lang.Class
    5  2.53% 35.40%   412256  212 17192688 9894  4126 [C
    6  1.88% 37.27%   305376 5849 1294096 20471   218 [C
    7  1.77% 39.04%   287280 1890  567248 4819  1601 [C
    8  1.68% 40.72%   273632 5787  273632 5787  3236 [I
    9  1.39% 42.11%   226544 3867 1729168 39226  1666 [C
   10  1.31% 43.42%   213200   13 26453200 1613  3505 [C
   11  1.27% 44.69%   206448  391 10368864 19638   107 [B
   12  1.21% 45.90%   196800   12 6445200  393  4485 [C
   13  1.16% 47.06%   188784   23 6287328  766  4484 [B
   14  1.14% 48.20%   184888 4581  184888 4581 96688 [C
   15  1.01% 49.20%   164000   10  164000   10 96675 [C
   16  0.90% 50.10%   146104   93 6077328 4132  4128 [C
   17  0.89% 50.99%   144480 1806  144480 1806  1762
java.util.HashMap$Entry
   18  0.88% 51.87%   142648  582  146064  631     1 [C
   19  0.81% 52.68%   132168 5507  295728 12322   171 java.lang.String
   20  0.79% 53.46%   127992  671  127992  671   506 [B

The worst 4 come from the stack trace 3236, which is as follows:

TRACE 3236:
	java.lang.ClassLoader.defineClass0(ClassLoader.java:Native
method)
	java.lang.ClassLoader.defineClass(ClassLoader.java:537)
	
org.apache.tools.ant.loader.AntClassLoader2.defineClassFromData(AntClass
Loader2.java:77)
	
org.apache.tools.ant.AntClassLoader.getClassFromStream(AntClassLoader.ja
va:1074)

So I started suspecting that maybe class loading is an issue.

I re-run visualgc, and noticed that the # of classes loaded is around
18500, and this number grows constantly throughout the whole build. I
took this to mean that new classes are constantly being loaded, which
doesn't sound right given that the build is just running XJC with
different schemas (IOW, you would expect that a lot of classes to be
loaded initially, then after the 1st compilation it should finish
loading most of the classes.)


Then I rerun Ant with -verbose:class, and found out that the same class
(like com.sun.tools.xjc.ConsoleErrorReporter) is loaded 14 times. The
build scripts compile 1 dummy schema + 13 schemas so this seems to imply
that somehow a fresh set of XJC classes is loaded every time one invokes
the XJC task, which seems like a terrible waste.

Now that I start to suspect Ant for memory leak, I searched Apache
BugZilla for some information. And I found this:

   http://nagoya.apache.org/bugzilla/show_bug.cgi?id=8689

In particular see the comment "Mike Aizatsky 2004-03-31 09:42", which
says he doesn't see the clean up to AntClassLoader.

Unless the class loader is freed, no classes loaded from it can be
freed.
So this seems to explain why XJC classes are loaded so many times.  So
this bug seems to explain everything.

I'll leave it up to you to apply the suggested patch locally and see if
the problem goes away. Meanwhile, this seems like a restriction of Ant,
so I'm afraid there's nothing we can do about this.


This concludes my little journey. Let me know how you think.

regards,
--
Kohsuke Kawaguchi
Sun Microsystems                   kohsuke.kawaguchi@sun.com


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@jaxb.dev.java.net
For additional commands, e-mail: users-help@jaxb.dev.java.net






> -----Original Message-----
> From: Stefan Bodewig [mailto:bodewig@apache.org] 
> Sent: Thursday, June 03, 2004 12:13 AM
> To: user@ant.apache.org
> Subject: Re: OutOfMemory and bug #8689
> 
> On Wed, 2 Jun 2004, Lee Breisacher <LBreisacher@seagullsw.com> wrote:
> 
> > Bug 8689 describes a situation where ant can get an 
> OutOfMemory error.  
> > This is happening to us when we run a JAXB build which contains a 
> > number of separate JAXB task invocations.
> 
> Are you sure it is a classloader issue in your case?  If so, 
> can you reduce the number of classloader instances created?
> 
> OutOfMemory most of the time happens because tasks really 
> need big chunks of memory.  If your XSLT transformer eats up 
> a GByte for a given transformation, there isn't anything Ant could do.
> 
> Other cases are memory/resource leaks in the task's code 
> (like javac for some JDKs since Sun's compiler was leaking resources).
> 
> In Ant itself, <ant> and <antcall> are the source for most 
> problems and a leak in AntClassLoader would manifest itself 
> here.  If you are using lots of <antcall>s you should look 
> into eliminating them, they've probably become unneeded in 
> 1.6.x (see macrodef for many instances).
> 
> > The last proposed fix (from Mike Aizatsky on 2004-03-31) appears to 
> > have NOT been incorporated yet.
> 
> Since it isn't really a fix - as can be seen by the new 
> problems it introduces.
> 
> > In any case, can someone tell me the status of this bug and 
> potential 
> > fixes?
> 
> AFAIK nobody is actively working on it.
> 
> Stefan
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org For 
> additional commands, e-mail: user-help@ant.apache.org
> 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org


Mime
View raw message