db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anders Morken <ander...@stud.ntnu.no>
Subject Re: [jira] Created: (DERBY-801) Allow parallel access to data files.
Date Wed, 24 May 2006 07:37:29 GMT
Øystein Grøvlen (JIRA):
> Allow parallel access to data files.
> ------------------------------------
> 
>          Key: DERBY-801
>          URL: http://issues.apache.org/jira/browse/DERBY-801
>      Project: Derby
>         Type: Improvement
>   Components: Performance, Store  
>     Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.1.1, 10.1.1.2, 10.1.2.0, 10.1.2.1  
 
>  Environment: Any >     Reporter: Øystein Grøvlen
> 
> 
> Derby currently serializes accesses to a data file.  For example, the
> implementation of RAFContainer.readPage is as follows:
> 
>     synchronized (this) {  // 'this' is a FileContainer, i.e. a file object
>         fileData.seek(pageOffset);  // fileData is a RandomAccessFile
>         fileData.readFully(pageData, 0, pageSize);
>     }
> 
> I have experiemented with a patch where I have introduced several file
> descriptors (RandomAccessFile objects) per RAFContainer.  These are
> used for reading.  The principle is that when all readers are busy, a
> readPage request will create a new reader.  (There is a maximum number
> of readers.)  With this patch, throughput was improved by 50% on
> linux.  For more discussion on this, see
> 
> http://www.nabble.com/Derby-I-O-issues-during-checkpointing-t473523.html
> 
> The challenge with the suggested approach is to make a mechanism to
> limit the number of open file descpriptors.  Mike Matrigali has
> suggested to use the existing CacheManager infrastructure for this
> purpose.  For a discussion on that, see:
> 
> http://www.nabble.com/new-uses-for-basic-services-cache---looking-for-advice-t756863.html

I've played around a bit with a different approach - using the
FileChannel class from Java 1.4's new IO API. I've written a class
RAFContainer4 which extends RAFContainer and overrides the readPage and
writePage methods of that class to use read/write(ByteBuffer buf, long
postition) in FileChannel to access the container's file, without
synchronizing on the FileContainer during the read and write calls.

With a bit of hackery in BaseDataFileFactory#newContainerObject() this
class is then used instead of the regular RAFContainer on creation of
new RAFContainer objects when Derby runs in a 1.4+ JVM.

This approach gives the JVM and OS the opportunity to issue multiple
file operations concurrently, although we have no guarantees that this
will actually happen. This is JVM/OS dependent, but stracing the Sun
1.4.2_09 VM on Linux 2.6 shows that the VM now uses pread64()/pwrite64()
system calls instead of seek(), read() and write(). pread and pwrite
have similar semantics to the FileChannel#read/write(ByteBuffer buf,
long position) methods, and do not alter the file's seek() position, and
are supposed to be thread safe.

Of course only people running Derby on 1.4+ JVMs will have the
opportunity to benefit from this approach. As support for 1.3 is to be
deprecated this might not be much of an issue?

But anyway, I would like to see if this hack of mine actually works. I
see mentions of a "TPC-B like benchmark" in the threads Øystein links to
above, and wonder if that is something Sun internal, or if it's a
publicly available benchmark implementation that I can get my grubby
little paws on and try out this patch with? =)

Thanks,
-- 
Anders Morken

My opinions may have changed, but not the fact that I am right!

Mime
View raw message