db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oystein.Grov...@Sun.COM (Øystein Grøvlen)
Subject Re: [jira] Commented: (DERBY-733) Starvation in RAFContainer.readPage()
Date Mon, 19 Dec 2005 16:06:52 GMT
>>>>> "MM" == Mike Matrigali <mikem_app@sbcglobal.net> writes:

    MM> excellent, I look forward to your work on concurrent I/O.  I am likely
    MM> to not be on the list much for the next 2 weeks, so won't be able to
    MM> help much.  In thinking about this issue I was hoping that somehow
    MM> the current container cache could be enhanced to support more than
    MM> one open container per container.  Then one would automatically get
    MM> control over the open file resource across all containers, by setting
    MM> the currently supported "max" on the container pool.

I have thought about using the current cache implementation for
caching open file descriptors.  However, I have not considered using
the container cache for this.  Will this work?  Do not some of the
code assume that there is only one object for each container?  (E.g,
truncation, backup).

    MM> The challenge is that this would be a new concept for the basic services
    MM> cache implementation.  What we want is a cache that supports multiple
    MM> objects with the same key, and that returns an available one if another
    MM> one is "busy".  Also returns a newly opened one, if all are busy.  I
    MM> am going to start a thread on this, to see if any other help is
    MM> available.  If possible  I like  this approach  better than  having a
    MM> queue of open files per container  where it hard to control the growth
    MM> of one queue vs. the growth in another.

I agree with you that a common cache would be better.  However, as you
point out, it is not straight-forward to use the current cache
implementation.  My first thought was that it would be too much work
to extend the current caching framework for this use, but I will think
a bit more about this, and follow up in your thread on this.

    MM> On the checkpoint issue, I would not have a problem with changes to the
    MM> current mechanism  to do "rwd" type  sync I/O rather than  sync at end
    MM> (but we will have to support both until we don't have to support older
    MM> versions of JVM's).  I believe this is as close
    MM> to  "direct i/o"  as we  can get  from java  - if  you  mean something
    MM> different here let me know.  

Yes, this was what I was thinking of.  Using direct i/o should give
better performance on reads.  However, some configurations may see a
performance drop since they depend on the file system buffer for good
performance.  However, such applications should be able to reclaim the
performance by increasing the page cache to not depend on file system

    MM> The benefit is that I believe it will fix the checkpoint
    MM> flooding the I/O system problem.  The downside is that it will
    MM> cause total number of I/O's to increase in cases where the
    MM> derby block size is smaller than the filesystem/disk blocksize
    MM> -- assuming the OS currently converts our flood of multiple
    MM> async writes to the same file to a smaller number of bigger
    MM> I/O's.  

It will vary from file systems to file systems how this will be done.
It would be nice to do some experiments here.  Maybe someone could do
a study on direct i/o on different file systems.  (E.g., the
performance impact of different page sizes).

    MM> I think this trade off is fine for checkpoints. If
    MM> checkpoint efficiency is an issue, there are a number of other
    MM> ways to address it in the future.

I agree.  Multiple writer threads are one way to address this.


View raw message