Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (nike.apache.org: domain of ames.greg@gmail.com designates
 209.85.212.182 as permitted sender)
MIME-Version: 1.0
Sender: ames.greg@gmail.com
Date: Tue, 15 Jan 2013 12:05:58 -0500
Message-ID: 
 <CAHSOX_ATn=cB5Y=ziOvc6Dqc4fTFbK+Y3EH3+FU-Sw5_pfm-TQ@mail.gmail.com>
Subject: proc_pthread accept mutex + graceful restarts = race condition
From: Greg Ames <gregames@apache.org>
To: httpd <dev@httpd.apache.org>
Content-Type: multipart/alternative; boundary=047d7b86e0ca8d808a04d356c604

--047d7b86e0ca8d808a04d356c604
Content-Type: text/plain; charset=ISO-8859-1

see PR 49504 https://issues.apache.org/bugzilla/show_bug.cgi?id=49504 for
an excellent analysis with supporting traces.  There have been various
other PRs that haven't led to resolution, perhaps because it is easy to
circumvent via AcceptMutex sysvsem and is highly timing dependent.

The problem affects the worker and prefork MPMs.  Event has a "get out of
jail free" card - no accept mutex!  I've only heard of it on Solaris,
probably because APR_USE_PROC_PTHREAD_SERIALIZE is the default on Solaris.
 We have a recent report of somebody hitting the problem consistently using
worker 2.2.x on Solaris 10 x64 running virtualized.

Basically the pthread accept mutex lives in pconf but the lifetime of pconf
is not quite right.  In server/main.c we have

    for (;;) {
        apr_hook_deregister_all();
        apr_pool_clear(pconf);

A graceful restart causes the MPM to notify the child processes to shut
down.  The MPM then immediately exits and causes this loop to iterate. Then
the last generation's pconf is cleared and off we go with the new config.
 The problem is that there is no guarantee that the old generation
processes are done with the accept mutex when we clear pconf.

Event appears to be doing fine with no accept mutex.
 prefork definitely needs one.  Some of the old timers may remember back
when we were trying to stabilize 2.0 prefork enough to ship a beta, and
Brian B's pager was going off at night due to high load on apache.org.  The
bug was due to no accept serialization in prefork around the poll for
listening sockets.

That makes me wonder whether worker really needs an accept mutex... is it
more like Event or prefork in this regard?  Since ThreadsPerChild 100 or
higher is reasonable on most systems these days with the old linuxthreads
library dead and buried, with that tuning we would only see 1% or less of
the polling overhead seen on prefork if worker didn't use an accept mutex
at all.  There's a chance that performance would actually improve due to
eliminating the lock/unlock path length.  I'm thinking "AcceptMutex none"
might be a good first step for worker.

Then what about prefork?  Some people apparently still care about it.  I
don't think we can blame APR's default mutex choice on Solaris for the
problem.  Inserting a sleep() or otherwise blocking the server main loop
seems like it would make graceful restarts not-so-graceful.  One idea is to
move the accept mutex into pconf's parent pool so it stays around forever.
 The guy who did the detailed analysis of the problem suggested refcounting
the uses of the accept mutex so that its last user could do the cleanup.
 That strikes me as too complex.

Thoughts/comments/patches?

Greg

--047d7b86e0ca8d808a04d356c604
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">see PR 49504=A0<a href=3D"https://issues.apache.org/bugzil=
la/show_bug.cgi?id=3D49504">https://issues.apache.org/bugzilla/show_bug.cgi=
?id=3D49504</a>=A0for an excellent analysis with supporting traces. =A0Ther=
e have been various other PRs that haven&#39;t led to resolution, perhaps b=
ecause it is easy to circumvent via AcceptMutex sysvsem and is highly timin=
g dependent.=A0<div>
<br></div><div style>The problem affects the worker and prefork MPMs. =A0Ev=
ent has a &quot;get out of jail free&quot; card - no accept mutex! =A0I&#39=
;ve only heard of it on Solaris, probably because APR_USE_PROC_PTHREAD_SERI=
ALIZE is the default on Solaris. =A0We have a recent report of somebody hit=
ting the problem consistently using worker 2.2.x on Solaris 10 x64 running =
virtualized. =A0</div>
<div style><br></div><div style>Basically the pthread accept mutex lives in=
 pconf but the lifetime of pconf is not quite right. =A0In server/main.c we=
 have</div><div style><br></div><div style><div>=A0 =A0 for (;;) {</div><di=
v>
=A0 =A0 =A0 =A0 apr_hook_deregister_all();</div><div>=A0 =A0 =A0 =A0 apr_po=
ol_clear(pconf);</div><div><br></div><div style>A graceful restart causes t=
he MPM to notify the child processes to shut down. =A0The MPM then immediat=
ely exits and causes this loop to iterate. Then the last generation&#39;s p=
conf is cleared and off we go with the new config. =A0The problem is that t=
here is no guarantee that the old generation processes are done with the ac=
cept mutex when we clear pconf.</div>
<div style><br></div><div style>Event appears to be doing fine=A0with no ac=
cept mutex. =A0prefork=A0definitely=A0needs one. =A0Some of the old timers =
may remember back when we were trying to=A0stabilize=A02.0 prefork enough t=
o ship a beta, and Brian B&#39;s pager was going off at night due to high l=
oad on <a href=3D"http://apache.org">apache.org</a>. =A0The bug was due to =
no accept serialization in prefork around the poll for listening sockets. =
=A0 =A0</div>
<div><br></div><div style>That makes me wonder whether worker really needs =
an accept mutex... is it more like Event or prefork in this regard? =A0Sinc=
e ThreadsPerChild 100 or higher is reasonable on most systems these days wi=
th the old linuxthreads library dead and buried, with that tuning we would =
only see 1% or less of the polling overhead seen on prefork if worker didn&=
#39;t use an accept mutex at all. =A0There&#39;s a chance that performance =
would actually improve due to eliminating the lock/unlock path length. =A0I=
&#39;m thinking &quot;AcceptMutex none&quot; might be a good first step for=
 worker.</div>
<div style><br></div><div style>Then what about prefork? =A0Some people app=
arently still care about it. =A0I don&#39;t think we can blame APR&#39;s de=
fault mutex choice on Solaris for the problem. =A0Inserting a sleep() or ot=
herwise blocking the server main loop seems like it would make graceful res=
tarts not-so-graceful. =A0One idea is to move the accept mutex into pconf&#=
39;s parent pool so it stays around forever. =A0The guy who did the detaile=
d analysis of the problem suggested refcounting the uses of the accept mute=
x so that its last user could do the cleanup. =A0That strikes me as too com=
plex. =A0</div>
<div style><br></div><div style>Thoughts/comments/patches?</div><div style>=
<br></div><div style>Greg</div></div></div>

--047d7b86e0ca8d808a04d356c604--