db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Van Couvering <David.Vancouver...@Sun.COM>
Subject Re: more background threads
Date Wed, 06 Apr 2005 22:16:11 GMT


Daniel John Debrunner wrote:
> David Van Couvering wrote:
> 
> 
>>OK, I think I'm beginning to get my head around the background thread
>>design.  It seems to be used in a lightweight way by a number of various
>>components, but is most heavily used by the transaction manager to
>>handle post-commit work.
>>As you mentioned, the current design spreads the work across the user
>>threads if the background thread gets backed up, and I agree that
>>overall throughput may not be much improved by having multiple
>>background threads do the work. 
> 
> 
> The intention of the item on the to-do list was to reduce the number of
> threads not increase them :-). The issue is that if a Derby system is
> booted with 1,000 databases, then 1,000 backgound threads are created,
> all just to handle background work. The to-list item was intended change
> the DaemonFactory from being booted per-database to being booted
> per-system.  Thus a single backgound thread would handle the idle
> requests. Then, I agree, the DaemonFactory could be enhanced to support
> a handful of worker threads behind the interface, without the knowledge
> of the caller.

Yes, when I re-read the todo item this is what you are getting at.  I 
can see value in having a pool of threads handle background work, but 
you are right it would be an explosion of threads if you have a scenario 
where a single JVM is handling thousands or even hundreds of databases 
(say one per application in a large application server deployment).  I 
had been unconsciously thinking from the perspective that you had only 
one background thread per VM, not per database.  I agree it really 
doesn't make sense to add worker threads unless they are shared across 
databases.

This implies that whatever test I write to check the behavior out should 
probably mimic a multi-database environment, where connections are 
spread out across databases.  I could set up a mechanism where 
connections are made in a round-robin or random fashion across a 
configurable number of databases...  Does this make sense?

> 
> The work involved I think is more involved than just changing how the
> factory is booted, since at the moment the context is tied to the
> thread, the change would require the context be separated from the
> thread and set dynamically as work came in from each database.
> In the end there would be a DaemonFactory per system and a DaemonService
> per database.
>

OK...  I'm still trying to get clear on your concepts and terms -- how 
is a "system" defined?  I'm thinking "all of Derby" within a given VM?


> 
> 
>>While reading the code, I did encounter one thing that had me
>>concerned.  What I am calling the transaction manager
>>(impl.store.Xact.java) seems to me to be overstepping its
>>responsibilities:  it takes care of running work items when the work
>>item requires immediate attention or if the daemon thread is backed up. 
>>This logic is not made available to any of the other components that
>>make use of the daemon service.  One can imagine this logic being
>>refactored out and made generally available to all clients of the daemon
>>service.  Is there a specific reason for having this code in Xact.java?
> 
> 
> I think it was a quick change to address a customer specific issue. I
> agree that cleanup is needed here, I once started some code that tried
> to cleanup the Serviceable intefaces, so that there was one set of
> constants (something like DONE,REQUEUE,IMMEDIATE, USERSPACE) and thus
> similar state is not separated over the return values from
> performWork(), serviceASAP and serviceImmediately() and the value passed
> into the enqueue method).

Yes, that was another odd thing I noticed and decided I would get back 
to later to try to understand.  The exact semantics of what happens when 
with the current implementation is a bit complex...  I could see this 
project as an opportunity to try and simplify some of this as you have 
discussed.

> Currently it is not well defined what should happen if the serviceNow
> parameter passed to enqueue is in conflict with the returns from
> serviceNow or serviceImmediately(). You can see that some of the
> implementations of Serviceable implement the current methods
> inconsistently (from memory).
> 

I hadn't noticed that yet, but I would not be surprised...

> 
> Thus I imagined a single method on Serviceable that indicated how the
> operation wanted to be serviced, make performWork void and don't have a
> serviceNow on enqueue.
> 

Yes, that does sound nice.

Before I spend a lot more time on this: are there other things that are 
reasonable in scale and scope you could really use my help on that are 
generally felt to be of higher priority?  I don't need to spend a lot of 
time on this if it currently works pretty well and is not causing a lot 
of pain or problems...

Thanks,

David

> 
> Dan.
> 

Mime
View raw message