Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: pass (asf.osuosl.org: local policy)
Date: Wed, 24 May 2006 09:37:29 +0200
From: Anders Morken <andersmo@stud.ntnu.no>
To: derby-dev@db.apache.org
Subject: Re: [jira] Created: (DERBY-801) Allow parallel access to data files.
Message-ID: <20060524073729.GA5933@stud.ntnu.no>
Reply-To: derby-dev@db.apache.org
References: <379656011.1136803530096.JavaMail.jira@ajax.apache.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <379656011.1136803530096.JavaMail.jira@ajax.apache.org>
User-Agent: Mutt/1.5.9i

�ystein Gr�vlen (JIRA):
> Allow parallel access to data files.
> ------------------------------------
> 
>          Key: DERBY-801
>          URL: http://issues.apache.org/jira/browse/DERBY-801
>      Project: Derby
>         Type: Improvement
>   Components: Performance, Store  
>     Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.1.1, 10.1.1.2, 10.1.2.0, 10.1.2.1    
>  Environment: Any >     Reporter: �ystein Gr�vlen
> 
> 
> Derby currently serializes accesses to a data file.  For example, the
> implementation of RAFContainer.readPage is as follows:
> 
>     synchronized (this) {  // 'this' is a FileContainer, i.e. a file object
>         fileData.seek(pageOffset);  // fileData is a RandomAccessFile
>         fileData.readFully(pageData, 0, pageSize);
>     }
> 
> I have experiemented with a patch where I have introduced several file
> descriptors (RandomAccessFile objects) per RAFContainer.  These are
> used for reading.  The principle is that when all readers are busy, a
> readPage request will create a new reader.  (There is a maximum number
> of readers.)  With this patch, throughput was improved by 50% on
> linux.  For more discussion on this, see
> 
> http://www.nabble.com/Derby-I-O-issues-during-checkpointing-t473523.html
> 
> The challenge with the suggested approach is to make a mechanism to
> limit the number of open file descpriptors.  Mike Matrigali has
> suggested to use the existing CacheManager infrastructure for this
> purpose.  For a discussion on that, see:
> 
> http://www.nabble.com/new-uses-for-basic-services-cache---looking-for-advice-t756863.html

I've played around a bit with a different approach - using the
FileChannel class from Java 1.4's new IO API. I've written a class
RAFContainer4 which extends RAFContainer and overrides the readPage and
writePage methods of that class to use read/write(ByteBuffer buf, long
postition) in FileChannel to access the container's file, without
synchronizing on the FileContainer during the read and write calls.

With a bit of hackery in BaseDataFileFactory#newContainerObject() this
class is then used instead of the regular RAFContainer on creation of
new RAFContainer objects when Derby runs in a 1.4+ JVM.

This approach gives the JVM and OS the opportunity to issue multiple
file operations concurrently, although we have no guarantees that this
will actually happen. This is JVM/OS dependent, but stracing the Sun
1.4.2_09 VM on Linux 2.6 shows that the VM now uses pread64()/pwrite64()
system calls instead of seek(), read() and write(). pread and pwrite
have similar semantics to the FileChannel#read/write(ByteBuffer buf,
long position) methods, and do not alter the file's seek() position, and
are supposed to be thread safe.

Of course only people running Derby on 1.4+ JVMs will have the
opportunity to benefit from this approach. As support for 1.3 is to be
deprecated this might not be much of an issue?

But anyway, I would like to see if this hack of mine actually works. I
see mentions of a "TPC-B like benchmark" in the threads �ystein links to
above, and wonder if that is something Sun internal, or if it's a
publicly available benchmark implementation that I can get my grubby
little paws on and try out this patch with? =)

Thanks,
-- 
Anders Morken

My opinions may have changed, but not the fact that I am right!