db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Van Couvering <David.Vancouver...@Sun.COM>
Subject Re: more background threads
Date Fri, 01 Apr 2005 01:07:34 GMT
Hi, Mike, thanks for the response and very helpful overview.  At first 
blush it seems like the single daemon could easily be converted to a 
thread pool approach where work is posted to a "dispatcher" who grabs a 
thread and dispatches the work to it.  I say this without having yet 
looked at the code, but in the meantime any reasons why this obviously 
won't work would be much appreciated.

I can work on building up a test case that fills up the background 
thread so we can "prove" that whatever solution we come up with helps 
the system scale better.  I can post a test plan prior to actually 
creating the test to see if you all agree the test looks to be what we 
want it to be.
 
I can also look into testing Derby scalability on a 4-way or 8-way 
machine, I think some of these are available in our lab.  I would also 
like to do some testing on some of Sun's new multi-core chips, where you 
have 8 threads per core and 4-8 cores per CPU.  Derby seems to be 
well-suited to this architecture but it would be good to see if there 
are any gotchas.  Again, I would proposed these as plans first and get 
your feedback.

What protocol do I use to sort of "identify" this is a sub-project and 
track its progress?  Do I create a JIRA item labelled as an 
"improvement" and assign it to myself?

Thanks,

David

Mike Matrigali wrote:

>I have changed the subject, as I completely missed the original post
>which had something to do with adding Junit tests.
>
>I am not sure what is the right solution here, but getting a discussion
>going would be good.
>
>Currently a number of store actions are queued in "post commit" mode,
>which means they should be executed until after the transaction which
>queued them commits.  Currently there is one background thread which
>processes these, if it gets too full then the work is done by the actual
>thread which queued the work.   Most of the post commit work involves
>claiming space from deleted rows after their transaction commits.
>
>Going forward there is going to be a need for more background work.  I
>soon will be posting the first phase of work to allow for returning
>space back to the operating system, eventually it would be best if this
>work was also done in background, somehow automatically queued by the
>system.
>
>I would also recommend coming up with a usage scenario which shows a
>problem before coding up a solution.  I believe a test with lots of
>users doing insert and delete should eventually show the background task
>being bogged down -- but I am not sure if moving work to additional
>threads is much better than just spreading the work out across the
>existing user threads.
>
>The code for the current background thread can be found in:
>opensource/java/engine/org/apache/derby/impl/services/daemon
>
>An example of one of the unit of work put on the queue is in:
>opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java
>
>Dan is probably the person who most recently worked on this code, and
>should have some comments in this area.  He should be back active on the
>list early next week.
>
>Note another interesting area of research/coding would be to see how
>derby scales on larger number of processor machines.  Not much work has
>been done at all on machines with more than 2 processors.  The system
>has been designed from bottom up to be multi-threaded, but not much
>testing/monitoring has been done on 4 or more processor machines.   The
>following single threading points exist in derby:
>    o each user query is executed by a single thread.
>    o the locking system in protected by a single java synchonization point.
>    o copying log records into the log is a single sync point
>    o finding a buffer in the buffer cache is a single sync point
>
>All of these seemed to be reasonable designs for 1, 2 and 4 way machines.
>
>/mikem
>
>
>David Van Couvering wrote:
>
>  
>
>>I noticed on the todo list there is a need to have more than one
>>background thread to enable better scalability with lots of client
>>connections.  I'm trying to find a way to gently work my way into doing
>>some work on Derby, and this seemed like a project of small enough scope
>>to get my feet wet.  Is there any background on this, or should I just
>>jump right in?  I didn't see any discussion of this on the list...
>>
>>Thanks,
>>
>>David
>>
>>    
>>

Mime
View raw message