db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satheesh Bandaram <sathe...@Sourcery.Org>
Subject Re: [jira] Created: (DERBY-646) In-memory backend storage support
Date Sun, 30 Oct 2005 23:59:13 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
It is pretty easy to pass information to use memory storage if URL
attributes were choosen. It is probably little bit more involved if use
of additional sub-protocols (memory:) is prefered. For attributes, it
doesn't even need a change to Derby client or Derby network server. But
the URL would have to look like:<br>
<br>
    'jdbc:derby://localhost:1527/mydb;<b>memoryStore=true</b>;' for
network client.<br>
<br>
Derby client simply passes any URL attributes that it doesn't know
about to server and to embedded driver. If embedded driver could be
changed to recognize this attribute and use new storage model, then it
would be available to network client and embedded clients this way.
This also makes it easy to set this as additional Datasource property,
if wanted.<br>
<br>
Satheesh<br>
<br>
Mike Matrigali wrote:<br>
<blockquote cite="mid4364E8D3.9030702@sbcglobal.net" type="cite">though
as you say it would be much more convenient to determine at connection
time.  I will let the network server experts comment on how best to get
the information from the client to the server.
  <br>
  <br>
  <br>
Stephen Fitch wrote:
  <br>
  <blockquote type="cite">Hi Mike,
    <br>
    <br>
the issue I'm having is I can't find a way to tell the network server
what StorageFactory to use from the network client driver.
    <br>
    <br>
On the embedded side of things, I just define a new subsubprotocol when
I start java with:
    <br>
-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory
    <br>
(alternatively I can change some of the engine code and register it as
a persistent service which results in the same issue)
    <br>
    <br>
then tell it to use that StorageFactory when I connect to the database,
ie:
    <br>
    <br>
jdbc:derby:memory:dbname;attributes
    <br>
    <br>
However, there does not appear to be an option to explicitly tell the
network server which storagefactory to use.
    <br>
    <br>
In the section entitled "Accessing the Network Server by using the
network client driver" at
<a class="moz-txt-link-freetext" href="http://db.apache.org/derby/docs/dev/adminguide/adminguide-single.html">http://db.apache.org/derby/docs/dev/adminguide/adminguide-single.html</a>,
it looks like the driver syntax does not support the above form of
connection URL. There does not appear to be an attribute for specifying
the storagefactory either.
    <br>
    <br>
I am looking for confirmation that this is the case.
    <br>
    <br>
If it is, I think the easiest way to add this functionality is another
connection attribute. This should avoid any code changes to the
drivers.
    <br>
    <br>
-------
    <br>
    <br>
As for my performance issues, I can tell the disk to turn off the write
cache, which it promptly ignores :) (Sound familiar?
<a class="moz-txt-link-freetext" href="http://weblogs.java.net/blog/davidvc/archive/2005/10/the_story_of_th.html">http://weblogs.java.net/blog/davidvc/archive/2005/10/the_story_of_th.html</a>)
    <br>
    <br>
Good suggestions to try durability=test mode and make the page cache
gigantic. I will test them versus my in-memory implementation.
    <br>
Thanks :)
    <br>
    <br>
I did figure out what was giving me the unexpectedly poor performance
increase while running in-memory.  A little error on my part.  I'm now
getting 2-3x better performance than Dir4StorageFactory :). Though this
number is entirely dependant on RAM speed, OS and disk speed so it will
vary between machines. My testbed was Gentoo Linux, kernel 2.6.13, JDK
1.4.2.08, 7200 RPM ATA hard drive w/8MB cache, 768MB of DDR266 RAM and
an AMD 2400+ CPU.
    <br>
    <br>
Thanks for your advice,
    <br>
    <br>
    <br>
    <br>
    <br>
Stephen Fitch
    <br>
Acadia University
    <br>
    <br>
Mike Matrigali wrote:
    <br>
    <br>
    <blockquote type="cite">I am not sure I understand your question
about network client/server and
      <br>
Storage factory.  The Network layer is way above the StorageFactory
      <br>
layer, so any change you make that makes an embedded client run in
memory will automatically also be used if you access it using the
network server interfaces (the in memory portion will always be on the
      <br>
server side of the client/server interface).
      <br>
      <br>
Your numbers don't surprise me if your disk is not really supporting
      <br>
synchronous I/O for log and data.  It may also be interesting to
compare
      <br>
your implementation numbers with running the existing server under
      <br>
durability=test mode, especially if you ever figure out how to get your
      <br>
disk to allow you to sync the log file.
      <br>
      <br>
The areas that are going to cause Derby to go slow:
      <br>
o sync of log file at commit time (write cache enabled disk disables
this sync)
      <br>
o sync of data files at checkpoint time (write cache disables this
sync)
      <br>
o sync of data files when db cache gets full (write cache disables, for
      <br>
  fair comparison you should make cache as big as database if you want
to compare with your in memory implementation)
      <br>
o read of uncached data from disk into cache - I guess this never
happens in your implementation as all data has to come about from
updates.
      <br>
      <br>
Stephen Fitch wrote:
      <br>
      <br>
      <blockquote type="cite">Hi Norbert,
        <br>
        <br>
I have it working in the trunk embedded client in gentoo linux under
jdk 1.4.2.08 but it should theoretically run under jdk 1.3.1 and any
recent release of derby. However, as far as I can tell the network
client/server doesn't support alternate implementations of
StorageFactory.  I'm still trying to track down if this is the case. 
If it is, another connection URL attribute may have to be added for
in-memory to work on the network side of things.
        <br>
        <br>
I'm having issues trying to test the performance as well since I'm
competing with derby's page cache, java.nio's and my hard drive write
cache (which I can't seem to turn off).
        <br>
        <br>
*VERY* preliminary numbers indicate a 10-20% performance increase on
inserts which is well below my expectations.  I'm going to code up some
tests for other operations today though.
        <br>
        <br>
If anyone's interested in trying the code out or going over it to look
for areas of improvement it would be much appreciated.
        <br>
        <br>
I suppose the best way to distro it would to email a svn diff patch and
source code for the new classes to the mailing list? I'm not looking to
get it added to svn though, as it still needs work.
        <br>
        <br>
Regards,
        <br>
        <br>
        <br>
        <br>
        <br>
Stephen Fitch
        <br>
Acadia University
        <br>
        <br>
        <br>
        <br>
        <br>
        <br>
        <br>
        <br>
        <br>
Toth-Gati Norbert wrote:
        <br>
        <br>
        <blockquote type="cite">Hi Stephen,
          <br>
          <br>
So now this means the support for in-memory storage is completed.
          <br>
I will give it a try. But glad you finally got to the end. Good job!
          <br>
          <br>
Regards,
          <br>
    Norbert
          <br>
          <br>
----- Original Message ----- From: "Stephen Fitch (JIRA)"
<a class="moz-txt-link-rfc2396E" href="mailto:derby-dev@db.apache.org">&lt;derby-dev@db.apache.org&gt;</a>
          <br>
To: <a class="moz-txt-link-rfc2396E" href="mailto:derby-dev@db.apache.org">&lt;derby-dev@db.apache.org&gt;</a>
          <br>
Sent: Tuesday, October 25, 2005 8:33 PM
          <br>
Subject: [jira] Created: (DERBY-646) In-memory backend storage support
          <br>
          <br>
          <br>
          <blockquote type="cite">In-memory backend storage support
            <br>
---------------------------------
            <br>
            <br>
        Key: DERBY-646
            <br>
        URL: <a class="moz-txt-link-freetext" href="http://issues.apache.org/jira/browse/DERBY-646">http://issues.apache.org/jira/browse/DERBY-646</a>
            <br>
    Project: Derby
            <br>
       Type: New Feature
            <br>
 Components: Store
            <br>
Environment: All
            <br>
   Reporter: Stephen Fitch
            <br>
            <br>
            <br>
To allow creation and modification of databases in-memory without
requiring disk access or space to store the database.
            <br>
            <br>
-- <br>
This message is automatically generated by JIRA.
            <br>
-
            <br>
If you think it was sent incorrectly contact one of the administrators:
            <br>
  <a class="moz-txt-link-freetext" href="http://issues.apache.org/jira/secure/Administrators.jspa">http://issues.apache.org/jira/secure/Administrators.jspa</a>
            <br>
-
            <br>
For more information on JIRA, see:
            <br>
  <a class="moz-txt-link-freetext" href="http://www.atlassian.com/software/jira">http://www.atlassian.com/software/jira</a>
            <br>
            <br>
          </blockquote>
          <br>
          <br>
        </blockquote>
        <br>
        <br>
      </blockquote>
      <br>
      <br>
    </blockquote>
    <br>
    <br>
  </blockquote>
  <br>
  <br>
  <br>
</blockquote>
</body>
</html>


Mime
View raw message