db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: more background threads
Date Thu, 31 Mar 2005 20:21:26 GMT
I have changed the subject, as I completely missed the original post
which had something to do with adding Junit tests.

I am not sure what is the right solution here, but getting a discussion
going would be good.

Currently a number of store actions are queued in "post commit" mode,
which means they should be executed until after the transaction which
queued them commits.  Currently there is one background thread which
processes these, if it gets too full then the work is done by the actual
thread which queued the work.   Most of the post commit work involves
claiming space from deleted rows after their transaction commits.

Going forward there is going to be a need for more background work.  I
soon will be posting the first phase of work to allow for returning
space back to the operating system, eventually it would be best if this
work was also done in background, somehow automatically queued by the
system.

I would also recommend coming up with a usage scenario which shows a
problem before coding up a solution.  I believe a test with lots of
users doing insert and delete should eventually show the background task
being bogged down -- but I am not sure if moving work to additional
threads is much better than just spreading the work out across the
existing user threads.

The code for the current background thread can be found in:
opensource/java/engine/org/apache/derby/impl/services/daemon

An example of one of the unit of work put on the queue is in:
opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java

Dan is probably the person who most recently worked on this code, and
should have some comments in this area.  He should be back active on the
list early next week.

Note another interesting area of research/coding would be to see how
derby scales on larger number of processor machines.  Not much work has
been done at all on machines with more than 2 processors.  The system
has been designed from bottom up to be multi-threaded, but not much
testing/monitoring has been done on 4 or more processor machines.   The
following single threading points exist in derby:
    o each user query is executed by a single thread.
    o the locking system in protected by a single java synchonization point.
    o copying log records into the log is a single sync point
    o finding a buffer in the buffer cache is a single sync point

All of these seemed to be reasonable designs for 1, 2 and 4 way machines.

/mikem


David Van Couvering wrote:

> I noticed on the todo list there is a need to have more than one
> background thread to enable better scalability with lots of client
> connections.  I'm trying to find a way to gently work my way into doing
> some work on Derby, and this seemed like a project of small enough scope
> to get my feet wet.  Is there any background on this, or should I just
> jump right in?  I didn't see any discussion of this on the list...
> 
> Thanks,
> 
> David
> 

Mime
View raw message