Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: neutral (asf.osuosl.org: local policy)
Message-ID: <44D2313A.7090903@apache.org>
Date: Thu, 03 Aug 2006 10:24:10 -0700
From: Daniel John Debrunner <djd@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
 rv:1.7.3) Gecko/20040910
MIME-Version: 1.0
To: derby-dev@db.apache.org
Subject: Re: Startup time for Derby
References: <44C955B7.2090502@sun.com>
 <54ac72d70608022151k3d49160fid2777c2132e92a0e@mail.gmail.com>
In-Reply-To: <54ac72d70608022151k3d49160fid2777c2132e92a0e@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Andrew McIntyre wrote:

> On 7/27/06, David Van Couvering <David.Vancouvering@sun.com> wrote:
> 
>> I am here at OSCON and a user of Derby was complaining rather
>> energetically to me at the cost of startup time for Derby.  He said this
>> is a real problem for running unit tests, as this is compounded by
>> running multiple tests, each one starting up a new database.
> 
> 
> This is something that has come up in the past and I just had a random
> thought regarding this.
> 
> The structure of a new, properly shutdown database has to be static
> for a specific version of Derby, right? (ok, almost, see locale note
> below)  So, what if we packaged a new database in a jar file,
> derbynewdb.jar, along with its exact version info, and when a
> connection URL with the create attribute is passed, the engine checks
> that newdb.jar is available and that the version of the newdb jar file
> in the classpath is identical. If so, the engine expands the new,
> empty database from the jar into the local filesystem instead of
> creating all of the system tables from scratch. If the version
> mismatches or the newdb.jar isn't there (or is incomplete?), we create
> a new db from scratch the regular way. A new db compressed into a jar
> file is only 54K for 10.2, not so bad for (re)deployment.
> 
> Someone has to have thought of this before, so I'm wondering what the
> issues are here. One problem that I can think of off-hand would be
> that a database is created with the JVM's current locale as the
> database locale. So it would be necessary to have a way to update the
> database locale after the expansion of the new db from the jar file,
> maybe a system procedure that needs to be called after the expansion
> of the new database from the jar. Another is what if there is a
> failure while uncompressing the db, what do you do on an attempt to
> reconnect? There would need to be a way to verify that you're working
> with a full deck of cards, as it were, but such a verification still
> seems like it would be less heavy-weight than creating a new db from
> scratch.
> 
> Anyway, the idea passed through my head, so I thought I'd throw it out
> there...

I think a useful first step is the ability to create a database from
another database, regardless of its stored form. Restore does this
today, but is limited to reading files. If that was expanded to handle
other stored forms (through the StorageFactory) then the base underlying
technology would exist for what you want, and is more generalized to
allow applications to copy pre-populated (schema and data) databases.

However, to have a existing empty template database that works in all
situations may be more effort than it worth. Apart from the locale
issue, the owner of database objects becomes important with SQL
authorization which will become the default at some point in the future.
I'm concerned that more and more code will have to exist to patch up
this template database. The current Derby scheme of easy to create new
database was in part reaction to my experience with another database
system which used the template scheme.

I think one issue is that the switch to a log that syncs to disk on
every write has slowed database creation time and has slowed performance
in other areas (namely insertion of large BLOBs/CLOBs). Previously the
log was marked as don't sync during db create, now that flag has no
effect and so the database creation has to wait for every log write. I
think this is something that should be addressed.

Dan.