httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Bloom <...@raleigh.ibm.com>
Subject fcntl problem in apache-apr.
Date Mon, 12 Apr 1999 12:24:51 GMT

As promised last night, here is an in-depth analysis of the fcntl problem
in hybrid apache.  I will also mention the two solutions we have found,
and their positive and negative points.  In an effort to be unbiased, I
will not mention which method I support.  :)

The basic problem, is that each socket has it's own fcntl lock file.  We 
use this file to ensure that only one process is accepting at any one
time.  The problem occurs whenever a process must go away, either because
of a graceful restart or because MaxReqeustsPerChild has been hit.  If one
of the accept threads is sitting in fcntl when the process is supposed to
quit, it stays there until it gets the lock.  In our testing, we have seen
that most processes don't give up the lock easily.  What tends to happen,
is a process retains the lock until it does not have the threads to serve
any more requests, and then it surrenders the lock to the next process.

Because we want to make sure we serve any connections that we have
accepted, we cannot kill the process until each of the acceptor threads is
in a known state.  We also know, that once a thread leaves the fcntl call,
it will die off if the process has been told to go away.  The problem
manifests itself most noticably when there are multiple sockets, and at
least one of them is not used often.  In this case, all of the often used
sockets will exit gracefully, but the rarely used socket will have its
thread stuck in fcntl.

I hope everybody understands the basic layout of the problem.  Now for the
killer.  There is no good way to exit the fcntl call.

We already tried closing the file that is being locked, but that caused
some nasty bugs, so it isn't an option.

Getting a signal would do it, but some threading libraries don't react
well to signals.  So, we are left with these two options.

There are basically three states for accept threads.  Accepting, Queueing,
and Blocking (on the accept lock).  Because we have to make sure that all
of our accepted connections are served, the only time it is okay to kill a
thread is if it is in Blocking state.  I can go over the rationale for
that if anybody would like me to, but that is a different e-mail.  

1)  Issue a pthread_cancel or it's equivalent.  This would be done in such
a way, that the thread could only be canceled when it is in the fcntl.
pthread_cancel at least has the ability to control when a thread can and
cannot be killed.  This is how we make sure the thread is in Blcoking
state.

pluses:  The threading library does most of the work for us.

minuses:  Not necessarily portable (to both windows, and other threading
libraries.)  pthread_cancel does not have a great reputation, because it
doesn't free locks and semaphores that the thread has (however, since the 
process is exiting, it isn't a big issue).  It requires a pthread_join so
that we know each thread has actually exited (pthread_join also has
portablility questions)

2)  Keep track of accept thread states, and exit the program when they are
all in valid states.  We aren't actually exiting the program, instead we
are just telling the workers it is okay for them to die when they finish
serving their current request, or immediatley if there are no more
requests.

pluses:  Definately portable.

minuses:  We have to do the work ourselves.  This requires either busy
waiting, or we would have to implement a logarithmic sleep algorithm
(sleep first for one sec, then two, then four, then eight, etc.), until
all threads are dead or in blocking state.

These two solutions are very similar, and could most likely be abstracted
out and merged together in apr, but that is a brief explanation of the
problem, and the current solutions that are being contemplated.  

Ryan

_______________________________________________________________________
Ryan Bloom		rbb@raleigh.ibm.com
4205 S Miami Blvd	
RTP, NC 27709		It's a beautiful sight to see good dancers 
			doing simple steps.  It's a painful sight to
			see beginners doing complicated patterns.	


Mime
View raw message