httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <...@covalent.net>
Subject Re: Scoreboard redesign
Date Fri, 18 May 2001 15:32:45 GMT

> > I am veto'ing the side effect of not allowing mod_status to be enabled on
> > a restart.
>
> Then there is nothing to veto. That was a bad temporary hack suggested by me that
> led this whole discussion down a rat hole. Issue vaporized.

Yes, that issue is vaporized, but I still believe you are going down the
wrong path.  You are adding a lot of complexity that is just not
necessary.

> >            I dislike the complexity that this whole thing is adding.
> > Traversing an array for the current status of the server is very clean and
> > elegant.  I do not believe that a linked list will be as clean.  I am not
> > veto'ing the linked list, I just don't believe it will be as clean a
> > solution.
>
> You are quite right. A linked list is more complex than a fixed position array.
>
> > I would prefer to see a solution that allows processes to be created with
> > fewer threads than required.  In fact, that is probably the easiest
> > solution to the whole problem.  Let the child process handle it.
> >
> > My understanding of the overall problem to solve, is that it is possible
> > for a single thread in a child process to keep that process slot from
> > being used for a new child.
> >
> > Divorce the threads from the process, and this problem goes away.  For
> > example, take this scoreboard.
> >
> > PID     THREAD  STATUS          ...
> > 10      1       WAITING         ...
> > 10      2       WAITING         ...
> > 10      3       BUSY            ...
>           (^--- Table 1)
>
> And what of the case where you have 50 threads and 7, 12, 32, 41, and 48
> are all busy...?

So what, this model extends to solve that problem.  The child process was
started, and it watches the scoreboard.  When one of the old threads dies,
the child process can create the new thread.  This puts the onus
completely on the child process to start itself.  The parent process
creates child processes, and then gets out of the loop until it wants to
kill of a child process.  This also automatically keeps us paying
attention to MaxClients.  Using a linked list doesn't have that benefit.
In the model I am suggesting, we never create too many threads, because
each child process has a corresponding slot in the scoreboard, and that
slot in the scoreboard has a maximum number of threads.  It doesn't matter
if the threads are used by an old child process, or a new one.  We can't
ever have more threads than MaxClients, regardless of the state of those
threads.

> > Now, assume we get a graceful restart request, the scoreboard goes first
> > to:
> >
> > PID     THREAD  STATUS          ...
> > 10              STOPING         ...
> > 10              STOPING         ...
> > 10      3       BUSY            ...
> >
> > This one child process could be busy for a while, but why not allow a new
> > child process to start, so that the scoreboard becomes:
> >
> > PID     THREAD  STATUS          ...
> > 11      1       STARTING        ...
> > 11      2       STARTING        ...
> > 10      3       BUSY            ...
>
> ...will you have the "10  X  Busy ..." scattered throughout the section
> for 7, 12, 32, 41, and 48? This seems like a more confusing and complex
> solution to me than a simple linked list design and seems more fraught
> with potential problems.

How is this complex?  The psuedo-code is simple enough:

    for (i=0; i < ap_threads_per_child; i++) {
        if scoreboard[my_child_num][i].state == BUSY
	    skip_this_one_for_now;
	else
	    apr_thread_create()
	    threads_created++;
    }

    /* all threads that could be started immediately have been, loop
     * and create the new ones */
    while (1) {

	/* make sure we haven't been told to die */
	if told_to_die
            kill_this_thread;

        for (i=0; i < ap_threads_per_child; i++) {
            if scoreboard[my_child_num][i].state != DEAD
		/* this process is still being used, either by this
		 * process, or the old one. */
	        skip_this_one_for_now;
    	    else
	        apr_thread_create()
		threads_created++;
        }
	if threads_created == threads_per_process
	    kill this thread;
        sleep 1 second;
    }


Done.  This should take much less time and be less error prone than a
brand new linked list mechanism.  Plus, it is far easier to debug, and
requires absolutely no locking.  The only place locking might be used, is
to ensure that only one process is creating new threads at one time, but
that is easily removed by only allowing one process to have a thread in
the thread_creation loop at a time.  This is done by having a mechanism in
the kill_this_thread step that signals the parent and says "I'm not
creating any more threads", once that message is received by the parent,
it can create the new child process.

> > The child can keep track of how many threads have already been started,
> > and once the last thread of PID 10 goes away, it can create the final
> > thread, so that the scoreboard becomes
> >
> > PID     THREAD  STATUS          ...
> > 11      1       WAITING         ...
> > 11      2       WAITING         ...
> > 11      3       STARTING        ...
> >
> > Now, we keep the array design, which allows for a very simple
> > implementation that does not require any locking, and we get the ability
> > to do a real restart.  I do not believe that it will be possible to create
> > a linked list implementation without locking, because you will need to
> > traverse the list, and elements will be allocated and freed on the fly.
>
> Yes locking is required with a linked list. The locking is only done when
> processes/workers enter or leave (which is not that often in cpu terms).
> The problem that we are trying to resolve is the case where, in the threaded
> mpm, each process has a small number of workers left finishing long requests.
> In your array based solution you could end up with N processes occupying the
> same fixed indexed space. For example, if threads_per_child = 5 you could get
>
> PID    THREAD  STATUS    ...
> 10     1       BUSY      ...
> 11     2       BUSY      ...
> 12     3       BUSY      ...
> 13     4       BUSY      ...
> 14     5       BUSY      ...

Yep, you could.  Your linked list has the same potential problem, but you
have the parent handling it all.  I am saying that solution will be much
more complex.  In this example, each child process is responsible for
cleaning itself up, and for creating the new threads, plus it is very easy
for our process to adjust to changes.

If the thread in PID 11 dies first, then PID 14 can create a new thread
for thst slot.  If PID 13's thread dies first, then PID 14 just takes over
that slot.

> In the linked list design, by returning the finished workers to the list
> you can start a new process any time that you get threads_per_process number
> of elements on the free list. This seems much easier to implement and understand.
> You would not end up with an example like table 1 above because the waiting
> workers would have finished up and exited on their own (when max_requests_per_child
> is hit) or would be reclaimed when the restart happens. So you would have:
>
> PID    THREAD   STATUS    ...
> 10     3        BUSY      ...
>
> Later you might end up with
>
> PID    THREAD   STATUS    ...
> 10     3        BUSY      ...      (Still being cleaned up...)
>
> 11     2        BUSY      ...      (Still being cleaned up...)
>
> 13     4        BUSY      ...      (Still being cleaned up...)
>
> 15     1        WAITING   ...      (Currently processing requests...)
> 15     2        BUSY      ...
> 15     3        BUSY      ...
> 15     4        BUSY      ...
> 15     5        WAITING   ...

This example violated the rule of creating too many threads.  In this
model, you have one process slot, being used by PIDs 10, 11, 13, and 15.
That one process slot gets 5 working threads, regardless of state.  But,
you have 8 threads.

> > The only addition to the code needed to implement this design, is a second
> > table, private to the parent, with the same number of elements as the
> > scoreboard.  This array will be used to store the pids of processes that
> > are actually dying.  Because the parent is the only thing to modify the
> > array, there is no locking involved, and it allows, for two different
> > processes to share one process space in the scoreboard, assuming that one
> > of the processes is trying to die.
>
> This seems like it (possibly) avoids locking, which shouldn't happen that often,
> for a more complex system of multiple tables tracking co-existence of multiple
> (potentially up to threads_per_process) processes within the same fixed
> scoreboard space. The list, even with its locks, seems simpler to me.

Go ahead and write the code for your model.  I will write the code for
mine.  We can both post to the list, and let the list decide which makes
more sense.

I believe that once you try to solve all the problems with linked lists
and shared memory, and all the problems with not creating too many
threads, your model will be far more complex to understand and to debug.
I am perfectly willing to be proven wrong, but I have given my
current opinion here.  I should have an implementation for my model some
time this weekend (assuming the house doesn't need a lot of work).

Ryan

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------




Mime
View raw message