httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul White" <>
Subject RE: mystery solved... perhaps.
Date Fri, 23 Jan 2004 04:57:28 GMT
A suggestion:

I'm not very sure what platform your on, but I'm assuming it's a unix
type of platform.  You can use 'netstat -ap' to find the socket (by
searching for its PID) that the httpd child is using to write its log
data, and see if its full.  The output will give you two counters: The
current send buffer size, and the current receive buffer size.  If your
send buffer size is maxed out, then that's definitely the issue.

Hope this helps...

-----Original Message-----
From: Ben Collins-Sussman [] 
Sent: Thursday, January 22, 2004 3:03 PM
Subject: mystery solved... perhaps.

After a couple days of research (and talking with Ed Korthof), I think
the mysterious bug is solved.  Recall the reproduction recipe:

  * start a long running 'svn import'
  * run 'apachectl graceful'
  * a few seconds later, httpd just hangs.

It looks like the main bug is with the 'rotatelogs' process that comes
with apache.  It has nothing to do with SSL at all -- it's
reproducible over HTTP, and even without subversion.

The theory (as I understand it) is this: if you set up your httpd.conf
appropriately, the httpd parent process launches a 'rotatelogs' child
process, along with N other httpd child processes.  All of the httpd
children keep write-pipes open to the rotatelogs process, and write log
data into their pipes.  'rotatelogs' has the job of reading these pipes
and spewing the data into appropriate files, creating new logfiles when

Here's what Ed Korthof thinks is happening:

  * the svn client (using neon) opens a long-lived connection to do a
    commit.  Using 'keepalive', it sends a huge number of PUT and
    PROPPATCH request over one connection to a single httpd child.

  * when the 'graceful' signal hits, httpd children wait for their
    current connection to close, then exit.  Meanwhile, the httpd
    parent spawns a new "generation" of httpd children.  Obviously,
    the httpd child servicing the svn commit sticks around a very long
    time, because svn doesn't hang up until it's done sending

  * for some reason, the 'rotatelogs' process dies.  It's not clear
    whether it's responding to a signal, or if the httpd parent is
    killing it, or what.  A new 'rotatelogs' takes its place, with new
    httpd children connecting to it.  Meanwhile, the "old" httpd child
    continues to service svn, and continues to write logdata to a dead
    pipe... there's nobody reading data from the pipe on the other end

  * Eventually the pipe fills up, and the httpd child just hangs
    trying to write to it.

I think this theory is true, for a few reasons:

  * every time I run 'strace -p PID' on the frozen httpd, it claims to
    writing logdata.   gdb confirms this as well.

  * edk is able to reproduce the problem without subversion, simply by
    hand-typing HTTP requests chained together by a Keep-Alive header.

  * 'svn import' claims to have received 'success' repsonses on about
    20 more files beyond what accesslog shows, implying a pipe-backup.

  * The clincher: in all my testing on different platforms (7 or 8
    different setups) this bug is reproducible *every* time httpd.conf
    is using 'rotatelogs', and the bug vanishes when I stop using

Final analysis: 

    This looks like some kind of bug in Apache itself, not
    related to SSL or Subversion at all... it looks like a bug in the
    interaction between 'rotatelogs' process and clients that use

Any comments or thoughts?

View raw message