db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Knut Anders Hatlen (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-2911) Implement a buffer manager using java.util.concurrent classes
Date Thu, 08 Nov 2007 13:44:50 GMT

     [ https://issues.apache.org/jira/browse/DERBY-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Knut Anders Hatlen updated DERBY-2911:

    Attachment: cleaner.diff

I have experimented a little more with the mentioned update load on a
large (~16GB) database. What I tried, was to change the rotateClock()
method in the new buffer manager so that when it found a dirty and
not-recently used page, it put the page in a queue which the
background writer would process, and go to the next page without
waiting for the previous page to be written. If the queue grew larger
than 1/10 of the maximum cache size (because the background thread
couldn't clean the pages fast enough to keep up with the user
threads), the user thread would however clean the page itself instead
of putting the page in the queue.

The attached patch (cleaner.diff) shows this approach. (Note that this
patch is only for testing. It doesn't actually use Derby's daemon
thread, but a separate thread that just sits and listens to an
ArrayBlockingQueue.) I tested it with the update load and page cache
size 1000, 10000 and 100000, and it seemed to have as good as or
better throughput than the old buffer manager in all of these
tests. (Graphs showing the results are contained in the cleaner.tar

To me this sounds like a good way to use the background thread, as it
offloads the user threads so that they don't need to wait for
potentially slow I/O operations. One might of course say that it's not
ideal, as it cleans pages right behind the clock hand instead of right
in front of it. However, the pages that are cleaned have not been used
during the last full round on the clock, so it is less likely that
they will be used on the next round as well, and therefore they'll
probably stay eligible for eviction. Also, if the background cleaner
works on pages right in front of the hand, it is quite likely that if
it finds a page to write, the user threads will catch up with it and
pass it while it's performing the I/O operation, so that it
effectively ends up cleaning behind the clock hand anyway.

If there are no objections or other good ideas, I will try to use the
approach in cleaner.diff and integrate it with Derby's own background

> Implement a buffer manager using java.util.concurrent classes
> -------------------------------------------------------------
>                 Key: DERBY-2911
>                 URL: https://issues.apache.org/jira/browse/DERBY-2911
>             Project: Derby
>          Issue Type: Improvement
>          Components: Performance, Services
>    Affects Versions:
>            Reporter: Knut Anders Hatlen
>            Assignee: Knut Anders Hatlen
>            Priority: Minor
>         Attachments: cleaner.diff, cleaner.tar, d2911-1.diff, d2911-1.stat, d2911-2.diff,
d2911-3.diff, d2911-4.diff, d2911-5.diff, d2911-6.diff, d2911-6.stat, d2911-7.diff, d2911-7a.diff,
d2911-entry-javadoc.diff, d2911-unused.diff, d2911-unused.stat, d2911perf.java, perftest6.pdf
> There are indications that the buffer manager is a bottleneck for some types of multi-user
load. For instance, Anders Morken wrote this in a comment on DERBY-1704: "With a separate
table and index for each thread (to remove latch contention and lock waits from the equation)
we (...) found that org.apache.derby.impl.services.cache.Clock.find()/release() caused about
5 times more contention than the synchronization in LockSet.lockObject() and LockSet.unlock().
That might be an indicator of where to apply the next push".
> It would be interesting to see the scalability and performance of a buffer manager which
exploits the concurrency utilities added in Java SE 5.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message