httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Fallon <davef....@gmail.com>
Subject [users@httpd] Apache getting stuck with all workers in a BUSY_READ state
Date Tue, 15 Jun 2010 17:02:30 GMT
Hi, I've been having problems with apache becoming unresponsive, and
was wondering if anyone had any suggestions on what the problem might
be. Basically, periodically, apache will get into a state where all
the workers are stuck reading:

Server Version: Apache
Server Built: Oct 21 2009 10:54:43
Current Time: Tuesday, 15-Jun-2010 07:57:30 PDT
Restart Time: Tuesday, 15-Jun-2010 06:37:33 PDT
Parent Server Generation: 0
Server uptime:  1 hour 19 minutes 57 seconds
Total accesses: 985801 - Total Traffic: 8.1 GB
CPU Usage: u644.89 s203.76 cu3994.75 cs0 - 101% CPU load
206 requests/sec - 1.7 MB/second - 8.6 kB/request
1593 requests currently being processed, 15 idle workers
RRRRRRRRRRRRRCRRRRKRRRRRRRRRRRRRRRRRRRKRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRCRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRCRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRCRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRCRRRRRRRRRRRRRRRRRRRRRRKRRRRRRRRRRRKRKRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRWRRRRKKCRRKRKRRRRKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRC
RRKKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRCKRRRRCCRRKRRRRRRRRRRRRRRRKRRRCRRRRRRRRRRRRRCCRRRCRRCRRR
RRRRRRRKKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRWRRRRRKRRRKRRRRRRRRRRRRRW
KKRRRRRRRKRRRRWRKRRRRRRRRRRRRRRRWRRRRRRRRRRRR___RRR__RR___R_____
WRR__RRRSS......................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

This is prior to complete failure - sometimes whatever's blocking gets
unblocked before it hits max clients, sometimes it doesn't. I'm
running apache 2.0.59 built with openssl 0.9.8n on AIX 6.1 with
prefork, and this is virtually all SSL traffic (pretty much everything
other than the scoreboard). A restart basically "fixes" the problem,
from the perspective that all the workers get killed and after the
initial thrashing of starting up new workers.

>From my understanding of the READ state above, everything above is
stuck in one of two broad categories:

 - A client made the TCP connection to the server, and is somewhere
between the tcp handshake and the end of the HTTP Request info. This
suggests it could be a network issue (something's hanging the
connections), or an openssl issue (the TLS/SSL negotiation is
slow/hanging), or...?
 - The request has been completed, but we're proxying to somewhere
else and waiting for a response from the proxy. This potentially
applies in this case, because we do have apache setup to proxy some
URLs to another server.

There's nothing in the access or error logs jumping out to correlate
with this problem either - There are MaxClient issues once it hits
that, of course, but nothing related to the BUSY_READ state.

 When having the problem, I've correlated the scoreboard with the
ps/lsof/netstat output, and the second case seems unlikely because I'm
not seeing any open connections to the server that apache is proxy'ing
to. It feels like there's some shared resource that all the apache
workers are trying to access, but I can't figure out what it might be.
Any suggestions on a solution, or how I might get more info out of
apache as to what it's doing while everyone's in the read state? Are
there other broad categories I'm missing as to why the workers might
be in the read state? Any further info I could provide to help anyone?
My next steps are to dive into the apache source further and see what
possible resources it could be blocking on, but I'm hoping someone
smarter than me already knows. :)


-- 
Dave Fallon

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message