Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Derby Development" <derby-dev@db.apache.org>
Received-SPF: neutral (hermes.apache.org: local policy)
Message-ID: <41BDD7D7.2070208@sbcglobal.net>
Date: Mon, 13 Dec 2004 09:56:39 -0800
From: Mike Matrigali <mikem_app@sbcglobal.net>
User-Agent: Mozilla Thunderbird 0.7.2 (Windows/20040707)
MIME-Version: 1.0
To: Derby Development <derby-dev@db.apache.org>
Subject: Re: alligned Vs Non Alligned log writes ..
References: <41B10688.6060202@Source-Zone.org>
In-Reply-To: <41B10688.6060202@Source-Zone.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

originally I assumed that alligned writing would increase the
log I/O throughput over the current use of buffered streaming
files.  My assumption was that on average every I/O we did would
partially cross at least 2 blocks and result in 2 I/O's for
every I/O we issued.

Given the performance results I now think that if the system
is doing a normal forced I/O when requested (ie. no write cache
enabled and no memory backed caching controller), then it is
not surprising we don't see much difference.  What I believe
is happening is that if 2 I/O's are indeed getting issued,
the OS/JVM is queing both I/O's basically at the same time and
then waits for both of them complete before returning.  In most
cases of recent disks my guess is that these 2 I/O's are likely
to go into a single sector and get forced in a single I/O given
the relatively small buffer sizes we use in the log.

If java ever provides a way to directly queue I/O straight from
our buffer to disk with no intermediate data copy then it may
be important to use a page based log scheme.  For now it looks
like the stream interfaces being used and the JVM's optimization
of those interfaces are working ok.  A nice property of the current
log which we don't take advantage of is that since it is a stream,
we could dynamically change the size of the block of data we
write, increasing it as log activity increases.  This sort of
happens with group commit where we will write less than a block
if a commit happens - but we never consider growing the buffers
bigger than the boot time size.

Has anyone used the advanced NIO interfaces in java?  We thought
that those should give us increased I/O throughput for things like
our log file.  When we looked at it about a year ago, our tests
didn't show the improvement we were looking for.  Anyone have
opinions on what is the fastest way to write blocks of data ranging
from 32k and down, and requiring sync to disk for each write?