Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: pass (asf.osuosl.org: local policy)
Date: Tue, 07 Mar 2006 16:54:43 +0100
From: John Embretsen <John.Embretsen@Sun.COM>
Subject: Re: [jira] Updated: (DERBY-210) Network Server will leak prepared
 statements if not explicitly closed by the user until the connection is
 closed
In-reply-to: <440D7D90.6010003@Sun.COM>
To: derby-dev@db.apache.org
Reply-to: derby-dev@db.apache.org
Message-id: <440DACC3.4040209@Sun.COM>
MIME-version: 1.0
Content-type: text/plain; charset=ISO-8859-1; format=flowed
Content-transfer-encoding: 7BIT
User-Agent: Thunderbird 1.5 (X11/20060113)
References: <1527603664.1113181887106.JavaMail.jira@ajax.apache.org>
 <851627339.1141405603295.JavaMail.jira@ajax.apache.org>
 <58ed70f50603061503l139220d3n91797d50f52ae7c9@mail.gmail.com>
 <440D7D90.6010003@Sun.COM>

John Embretsen wrote:

> Deepa Remesh wrote:
> 
>> Thanks John for uploading your repro. I have not been able to look at
>> it closely as jira seems to be having problems again. I was wondering
>> if you have the results from your DOTS test. Please post if you have
>> any updates.
> 
> Actually, the DOTS test failed, i.e. the network server hit an 
> OutOfMemoryError (PermGenSpace) after about 56 hours (not much longer 
> than without the patch), but I suspect that something is not right...
> 
> I get strange results from running my repro with the same jars, and 
> *sometimes* with freshly built jars even with your patch included. This 
> is not what I was seeing before the weekend. I am doing some detective 
> work to figure out what could be wrong with my environment (scripts, 
> classpaths, JVM settings, build process, patch, etc.). When I figure it 
> out, I will re-run the tests and post results.

I think I figured out what was causing the unpredictable results when running 
my repro:

I was (most of the time, both for DOTS and my repro) running with the 
following flags on the server JVM:

-Xms128m -Xmx128m -XX:PermSize=64m -XX:MaxPermSize=64m

These flags are setting both initial and max size of both heap and PermGen 
space, with initial size equal to max size. This was to ease debugging (I 
think it is easier to interpret memory usage graphs when I know the 
capacities of the different parts of the heap). Besides, people recommend 
setting initial and max heap size equal to each other for performance 
reasons, so I did the same for the permanent generation.

Anyway, I found out that *without* the flag "-XX:PermSize=64m" the test/repro 
"passed", while *with* the flag set, the server usually crashed at 
approximately 11000 statements. Here is my theory, based on observation of 
jstat output:

With my repro, derby generates classes faster than the VM is able to garbage 
collect in one go. According to PermGen usage logs (jstat), roughly 500k 
worth of bytecode is injected into the permanent generation per second (I 
guess mostly due to class generation since most statements sent from the 
client are unique).

So, when the initial PermGen size is smaller, i.e. not close to, the max 
size, the VM can increase PermGen capacity while garbage collecting old 
classes (or something like that). Perhaps there is some small delay before 
classes can be unloaded/GCed, I don't know.

I have successfully and predicably run my repro again, now without the 
"-XX:PermSize" flag set. Without patch5 for DERBY-210, the server hits an 
OutOfMemoryError after roughly 11000 statements. With patch5, the server is 
able to gc all parts of the heap, including the PermGen space (I have tested 
100000 statements so far, after I discovered this issue).

I have started a new DOTS test run, this time without setting the initial 
PermGen size. I am crossing my fingers and hoping this will improve the 
results of the DOTS test.

There could of course be other issues with my environment, so I still 
recommend that others try some of these tests/repros. Use whatever JVM 
settings you feel like, but I recommend not setting initial PermSize equal 
(or close) to max PermSize.


-- 
John