db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Thalamati <suresh.thalam...@gmail.com>
Subject Re: [jira] Commented: (DERBY-801) Allow parallel access to data files.
Date Mon, 18 Sep 2006 23:56:43 GMT
Anders Morken wrote:
> Suresh Thalamati (JIRA):
> 
>>    [ http://issues.apache.org/jira/browse/DERBY-801?page=comments#action_12434613
] 
>>            

<snip ..>


>>RAFContainerFactory.java
>>------------------------
>>
>>Logic in this new class seems to be deciding whether to load RafContainer.java or
>>the RafContainer4.java based on the JVM.  I am not sure, if  this logic is
>>necessary here. Did you consider using basic services to load the java classes 
>>specific to a JVM ?
> 
> 
> Can you do that? Never occured to me, I'll check it out. Thanks for the
> tip. Any pointers and tips would be welcome, but given enough time and
> source code perusal I'll probably figure it out on my own as well. From
> a quick look at modules.properties it looks pretty straightforward, at
> least if adding "J4" to a module name means "load this in Java 1.4 and
> up". =)

you are are the right track. I have used this mechanism long time ago, you
might need to  something like:

Replace the current data factory module entry
(:derby.module.rawStore.data.generic) with the following entries :

derby.module.rawStore.dataJ1=rg.apache.derby.impl.store.raw.data.BaseDataFileFactory
derby.env.jdk.dataJ1=1
derby.config.rawStore.dataJ1=derby

derby.module.rawStore.dataJ4=org.apache.derby.impl.store.raw.data.DataFileFactoryJava4
derby.env.jdk.dataJ4=4
derby.config.rawStore.dataJ4=derby

DataFileFactoryJava4 will be the new class extended from 
BaseDataFileFactory.




> 
> 
>>I think basic services has support to boot a specific factory implementation
>>based on the JVM using modules.properties. For example in the current
>>scenario, one can extend BaseDataFileFactory.java class to  
>>implement newContainerObject(), which will return the RafContainer4( ..).  add
>>the new class to modules.properties to boot only on versions >=jdk14.
>>
>>
>>

> 
>>4) I noticed, there is new encryption buffer created on every writePage() call,
>>   if the database is encrypted. This may cause jvm peak memory usage increase,
>>   when there is a checkpoint, if there are lot of dirty pages in the cache and
>>   if garbage collection is not happening fast enough.  I hope this does not
>>   lead to out of memory errors!
>>
>>   We may need to implement some kind of scheme, that will help in reuse of
>>   the encryption buffers. 
> 
> 
> This is a (classical) choice between concurrency and resource
> consumption.  On one hand, object creation and garbage collection from
> the eden space is pretty cheap on 1.4 and newer VMs, but we'll force
> additional GC runs... This is a complex subject, with many variables. =)
> 
> I suppose it could be interesting to do a little throughput testing with
> this patch and an encrypted database and see if there is any impact. I
> can't really say that I have any idea if recycling a single buffer or
> creating lots of buffers is the cheaper way...
> 

I agree, new jvms are smarter. This problem can be addressed later, if 
needed. On older jvms I used to run into out of memory problem when I 
do some thing like that.



> 
>>5) I am ok with readPage() and writePage() routines in RafContainer4.java. 
>>   just curious , if you considered implementing  new read/write..etc  calls in 
>>   the RafContainer4.java using file channel and just wrapper methods in the
>>   RafContainer.java using the existing random access file,  instead of 
>>   overriding readPage()/writePage() ...etc. 
> 
> 
> Uhm, I'm sorry, I'm not sure I understand what you're thinking of here?
> Could you elaborate a bit? =)

To avoid dealing with changes to two copies of readPage/writePage 
routines in
future fixes; I would have tried to abstract out just the code that is 
really
different in jdk14 vs jdk13. For example :


in RafContainer.java , implement :

writeBuffer( .....) {
   synchronized(this) {
      fileData.seek( ...)
      fileData.write( ..)
   }
}


and in RafContainer4.java, override that routine :

writeBuffer(....) {

  iochannel.write( ...)

}

similarly any other functionality you need.  I undestand, it adds an 
extra level to the call stack; my guess is that should not have major 
performance impact.

For all I know , it may not be that simple ; Different exceptions 
..etc. Please feel free to ignore this suggestion, if you think 
current approach is good enough.



> If we wanted to go even further we could consider using memory mapped IO
> - then the parts of Derby that modify data pages could access the memory
> mapped file directly and save some overhead, but that probably requires
> a bit of a rework of some interfaces and the code that modifies data
> pages. And then we have the case of large databases and 32-bit address
> spaces... =)
> 

It would be intersting to find, if that improves the performance further.


> 
>>6) Please file a JIRA  to enhance StorageFactory interfaces to support NIO. 
> 
> 
> There's already DERBY-262 in Jira, but it's a bit unspecific... Anything
> in particular you think needs to be added to the interfaces?

DERBY-262 is a very generic NIO entry. I think it would be better to 
file a separate JIRA entry that will describe the changes needed in 
the storage factory to implement what you are doing in this fix using 
the storage factory, instead of limiting the RAFContainer4.java usage 
only to storage factory that is based on the java.io.RandomAccessFile.


Thanks
/suresh




Mime
View raw message