db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: [jira] Commented: (DERBY-733) Starvation in RAFContainer.readPage()
Date Thu, 15 Dec 2005 01:07:56 GMT
Originally I would have preferred a separate loadable module approach, 
but I sent Bryan down that path in the lock manager and it seemed like
a daunting task to get the new file to build under jdk1.5.
And now I realize that it may require a derby level vote to require 
jdk1.5 to build the system. At the time I didn't realize we already
had reflection based calling of jdk1.5 stuff in the code.

I looked at the reflection callse in SQLDecimal.java and this didn't 
seem that different.
There is similar code in SQLDecimal.java, do you think that that code 
should have been in a separate module or is that a different case?

I now realize that one can still build separate modules using the
derby module system to load the appropriate code, and go ahead and use 
reflection in the separate module to localize the "if (java5)" kind of 
code.  I hadn't considered this or the use of derby latches when 
reviewing the fix.  I am sure it is not clear from the public interface 
description that the derby locking manager guarantees "fair" scheduling,
though I know it does (there are actually some cases for low priority 
background threads that it would not do that - but that is another 
story).

This module and a few others could use some improvements based on
some observations on the list:
o current user thread i/o sometimes causes user threads to be starved
o current checkpoint sometimes causes user threads to be starved
o current I/O is single threaded per file (a file maps to a table or an 
index)

It is clear that the single point of contention in RAFContainer is a 
problem that is showing up on bigger, more concurrent machines.   From
the evidence that has been coming on the list I think the following
needs to somehow be solved:

user thread initiated read
     o should be high priority and should be "fair" with other user 
  initiated reads.
     o These happen anytime a read of a row causes a cache miss.
     o Currently only one I/O operation to a file can happen at a time,
       could be big problem for some types of multi-threaded, highly
       concurrent low number of table apps.  I think the path here
       should be to increase the number of concurrent I/O's allowed
       to be outstanding by allowing each thread to have 1 (assuming
       sufficient open file resources).  100 outstanding I/O's to a 
single file may be overkill, but in java we can't know that the file is 
not actually 100 disks underneath.  The number of I/O's should grow
as the actual application load increases, note I still think max I/O's
should be tied to number of user threads, plus maybe a small number for
background processing.

user thread initiated write
      o same issues as user initiated read.
      o happens way less than read, as it should only happen on a cache
        miss that can't find a non-dirty page in the cache.  background
        cache cleaner should be keeping this from happening, though apps 
that only do updates and cause cache hits are worst case.

checkpoint initiated write:
      o sometimes too many checkpoints happen in too short a time.
      o needs an improved scheduling algorithm, currently just defaults
        to N number of bytes to the log file no matter what the speed of
        log writes are.
      o currently may flood the I/O system causing user reads/writes to
        stall - on some OS/JVM's this stall is amazing like ten's of 
seconds.
      o It is not important that checkpoints run fast, it is more 
important that it prodede methodically to conclusion while causing a 
little interuption to "real" work by user threads.  Various approaches
to this were discussed, but no patches yet.


Daniel John Debrunner (JIRA) wrote:
>     [ http://issues.apache.org/jira/browse/DERBY-733?page=comments#action_12360463 ]

> 
> Daniel John Debrunner commented on DERBY-733:
> ---------------------------------------------
> 
> The issue I have with this current patch is that is is localized to one use in RAFContainer,
when reading files.
> Most likely there are other locations where such a facility would be useful, especially
on the write for RAFContainer.
> Are we going to have similar if (java5) statements everywhere. The module api already
supports loading different
> code for different environments, I think that this functionality could be added to the
LockManager  or  maybe a separate
> module. This would be an improvement, but I think it would be a mistake to have similar
code to this patch in many
> areas of Derby. And yes, maybe we could work on improving Latch performance along these
lines.
> 
> 
>>Starvation in RAFContainer.readPage()
>>-------------------------------------
>>
>>         Key: DERBY-733
>>         URL: http://issues.apache.org/jira/browse/DERBY-733
>>     Project: Derby
>>        Type: Improvement
>>  Components: Performance, Store
>>    Versions: 10.2.0.0, 10.1.2.1, 10.1.3.0, 10.1.2.2
>> Environment: Solaris x86 and Linux with Sun JVM 1.5.0. Derby embedded and client/server.
>>    Reporter: Knut Anders Hatlen
>>    Assignee: Knut Anders Hatlen
>> Attachments: DERBY-733.diff
>>
>>When Derby is completely disk bound, threads might be starved in
>>RAFContainer.readPage(). This is a real problem when multiple clients
>>are repeatedly accessing one or a small number of large tables. In
>>cases like this, I have observed very high maximum response times
>>(several minutes in the worst cases) on simple transactions. The
>>average response time is not affected by this.
>>The starvation is caused by a synchronized block in
>>RAFContainer.readPage():
>>  synchronized (this) {
>>      fileData.seek(pageOffset);
>>      fileData.readFully(pageData, 0, pageSize);
>>  }
>>If many threads want to read pages from the same file, there will be a
>>long queue of threads waiting for this monitor. Since the Java
>>specification does not guarantee that threads waiting for monitors are
>>treated fairly, some threads might have to wait for a long time before
>>they get the monitor. (Usually, a couple of threads get full throughput
>>while the others have to wait.)
> 
> 


Mime
View raw message