httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Woodworth" <mirkp...@gmail.com>
Subject Re: mod_proxy race condition bug #37770
Date Fri, 16 May 2008 21:21:29 GMT
So I have some more information about this, but this time related to
having keepalives OFF in mod_proxy.

I tried using the "SetEnv proxy-nokeepalive 1" option in httpd.conf,
and it cleared up the proxy errors that I was having with an IIS
backend server, and it may have decreased the proxy errors with Apache
backend servers as well.

I tried again without the proxy-nokeepalive option, and the proxy
errors increased.  Put it back, they decreased dramatically again.

Turns out that mod_proxy always sends "Connection: Keep-Alive", but
with "proxy-nokeepalive 1", mod_proxy will always send "Connection:
Close".

The backends in my case are all responding with "Connection: Close",
so the backends aren't even allowing Keep-Alives, regardless of
whether mod_proxy sends "Connection: Close" or "Connection:
Keep-Alive".

My theory is that mod_proxy is incorrectly using a socket pool even
when the server responds with "Connection: Close" or when mod_proxy
itself sends "Connection: Close".  Most of the time it probably
realizes that the sockets in the pool are actually closed, but
sometimes not and it tries to use a dead socket.

What should be happening is that mod_proxy needs to NOT use a socket
pool if "Connection: Close" is in use.

However, I have not looked very deeply into the code to see if my
theory is accurate.  Could someone more familiar with the mod_proxy
code please fill me in?  Does mod_proxy always use a connection pool?

Thanks,
Adam


On Mon, May 12, 2008 at 4:15 PM, Adam Woodworth <mirkperl@gmail.com> wrote:
> I haven't looked at the code in mod_proxy to see how it handles the
> Keep-Alive header returned by the backend server, but what I'm seeing
> in this tcpdump I have that shows the proxy error happening is that
> the backend webserver is IIS, and it is not sending any Connection or
> Keep-Alive headers back to mod_proxy, even though mod_proxy sent a
> Connection: Keep-Alive header.  But, this is HTTP 1.1 -- is the server
> required to send back a Connection and Keep-Alive header?
>
> What I'm wondering is, if the server doesn't send back a Keep-Alive
> header that specifies the timeout and max requests (e.g., "Keep-Alive:
> timeout=15, max=500"), then does mod_proxy just default to an infinite
> (or very large) timeout for that connection?  And somehow it's not
> noticing that the server closed the connection?  What I see is that in
> a nearly 700 second long tcp dump, there are a few mod_proxy requests
> made to the backend w/o any SYN/SYN-ACK/ACK TCP handshaking happening
> before the GET packet, which means that Apache must think that the
> socket is still open -- i.e., it used the socket for a connection a
> bunch of minutes ago, and still thinks it's alive.  But I see this
> happening 600 seconds into the dump, without any other activity on the
> same socket prior to that in the dump, which leads me to think that
> Apache is just keeping the socket around for quite a long time.  But
> it surprises me that IIS hasn't closed the socket by then, so that by
> the time Apache tried to use it it would notice it was closed...hmmm,
> very strange.
>
> Adam
>
>
> On Mon, May 12, 2008 at 3:31 PM, Nick Kew <nick@webthing.com> wrote:
>> On Mon, 12 May 2008 13:52:18 -0400
>>  "Adam Woodworth" <mirkperl@gmail.com> wrote:
>>
>>  > Hi,
>>  >
>>  > I was wondering if anyone might have some more information on bug
>>  > #37770.  I've added a comment there recently, at the end of the bug
>>  > report, #83.
>>  >
>>  > https://issues.apache.org/bugzilla/show_bug.cgi?id=37770
>>
>>  If this is indeed a race condition biting you, it's possible
>>  you could fix it by setting a sufficiently low ttl value that
>>  the backend's ttl will be guaranteed to outlive it - depending
>>  of course on backend behaviour.  So if the backend has a
>>  15 second ttl, set 10 secs in the proxy, and the remaining
>>  5 secs provide a good buffer against a race condition.
>>
>>  Except that ttl doesn't work exactly as documented.
>>
>>  There's a similar issue with ttl and DBD database connections.
>>
>>  I've just posted on the subject to dev@apr ("apr_reslist
>>  semantics").  This could enable you to configure apache
>>  to avoid this problem without sacrificing backend keepalives
>>  altogether.
>>
>>  --
>>  Nick Kew
>>
>>  Application Development with Apache - the Apache Modules Book
>>  http://www.apachetutor.org/
>>
>

Mime
View raw message