db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Pendleton <bpendle...@amberpoint.com>
Subject Re: Managing many databases
Date Tue, 15 Apr 2008 20:11:42 GMT
> But I'm not sure how best to actually manage all these databases. I 
> suspect we will have on the order of 1000 databases in play, with maybe 
> 20 of those being actively used at a single busy time. 

When a database is not open it should be consuming no resources. There
is no global registry of databases in a particular Derby install; each
database is accessed independently, with its own set of in-memory data structures.

So, as long as you are careful to fully close and shutdown each database
when it is not in use, a carefully-written server app should be able
to alternate usage of a moderate-sized handful (20 seems well within reason)
of a much larger universe of latent databases on your filesystem (1000
seems well within reason here, too).

Give yourself lots of memory and lots of file descriptors. Use the most
modern JVM you have access to. And, if you're going to put thousands of
multi-gigabyte individual databases onto a single filesystem, you'll
need a multi-terabyte file system, so make sure you get a solid server
operating system that can handle that.

One caution: in recent history there have been a (small) number of bugs
involving Derby datastructures which were not 100% garbage collected
properly. Although those bugs have (we believe) been fixed, you might
encounter more of them, *plus* you need to be extremely careful in your
own application to ensure that you fully close all result sets, statements,
connections, and other similar JDBC objects and remove those objects
from all global hash tables and other collections, to ensure that when
you shut down one database, all of its resources can be safely cleaned
up by the JVM.

And set up a thorough test suite that explores these scalability
behaviors, and run both simulated tests and real live tests. You should
be able to run a simulated test fairly early in your development, by
simply fabricating a large number of synthetic databases and verifying
that your application works well in that environment. Modern JVMs have
superb profiling tools that you can use to verify the desired behavior.

Good luck, and keep us updated on how it goes!



View raw message