db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anders Morken <ander...@stud.ntnu.no>
Subject Re: out of memory when writing blobs
Date Wed, 14 Mar 2007 13:58:25 GMT
frederic barachant:
> I tried that, but i had the bug anyway.
> I tried your option again, using commandline, netbeans's VM options, ... 
> anywhere i could and it failed each time. (see end of the mail for the 
> stack trace so you can compare with yours, just in case...)
> I added this line:
> System.err.println("size:"+System.getProperty("derby.storage.pageCacheSize"));
> right before the class.forName() of the source i provided. I see the 
> right value printed, so it seems obvious that i am really setting that 
> value correctly.

Hm, I've tried with a few releases now, and the ones prior to 10.2.2
bomb with an OutOfMemoryError right away, while 10.2.2 (as well as a
build of the development trunk) easily churn through a few hundred
iterations when I use a small cache.

I'm running this on Sun's client VM, version 1.6.0-b105 on Linux, btw.

> Side note about the end of your response. I expected your cache, just as 
> any cache, to reference cached data using a java.lang.ref.SotfRererence. 
> Is there something i do not see that did prevent their use?
> At least, it would have reduced the OOME to the inevitable case when 
> everything referenced cannot be freed.

Nope, Derby's page cache is kind of possessive, it doesn't want any
buffers to suddenly disappear without having a final say itself. I
figure soft/weak references are something we could explore, though, we'd
just have to make sure that the garbage collector won't go and take any
of the dirty buffers, or any pages that are actively being used. (That
is, we'll need a mix of regular and soft references. =)

Possibly good news: It isn't unlikely that the cache manager will
receive some attention. There are a few known scalability issues with
the current implementation, and a Google Summer of Code project in 2006
explored a few ways to improve Derby's page cache in terms of
algorithms. Note that I'm not a regular Derby developer, and certainly
not in a position to guarantee that something will happen wrt the page
cache, I'm just a student who happens to play with Derby as part of my
studies. =)

PS. My small, slightly modified test case based on yours, successfully
churned through >1500 iterations of 6MB blob insertion with Derby in the classpath and a 100 page page cache while I wrote this
email. I moved the prepareStatement outside the loop, replaced your
copy/paste with a single while(true) {...} and added a counter to count
the number of loops but it's otherwise cut-and-pasted from what you

Anders Morken

My opinions may have changed, but not the fact that I am right!

View raw message