httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Derouin <tdero...@gmail.com>
Subject Re: [users@httpd] Apache 2.0.52 - stops responding
Date Sun, 03 Oct 2004 16:31:20 GMT
Hi Gary,

When the server goes down, we have about 100 or so established
connections, it seems there are several IPs listed, any given IP has
between 1 and 5 open connections to the server. It's odd, because last
night when we crashed, I wouldn't expect 100 people to be on the site
at such a low traffic time.

One of the technicians suggested we turn off KeepAlive, that seemed to
reduce the number of requests being processed in the server status
window, but we later experienced the same problem. It does seem like
there are a lot of established connections to resin (about 1300 -
which seems a excessive), does a connection to resin count as a
working thread, or are they in some pool? Why does the pool grow so
big, or is that just a resin setting?

I was seeing a bunch of "AcceptEx failed. Attempting to recover."
error logs (about 9k per day), so I disabled AcceptEx as described in
the faq, but it didn't seem to fix the problem of the server coming to
a halt.

It seems that, looking at the TCP/IP RFC (1122) and Microsoft's KB
article on TCP/IP Connection states (137984) that the amount of
close_wait states are evidence of an interruption in the closing
handshake. The client sends a FIN to the server, the server's socket
goes the the CLOSE_WAIT state, then the server sends an ACK and it's
own FIN to the client, and then the client sends back the ACK. So it
seems at least one of these three packets is getting lost, because the
server socket never goes to CLOSED state. The KB article gives a brief
example:

A socket application has been terminated, but Netstat reports the
socket in a CLOSE_WAIT state. This could indicate that the client
properly closed the connection (FIN has been sent), but the server
still has its socket open. This could be the result of one instance
(among all threads or processes) of the socket not being closed.

Is Apache holding on to closed sockets for some reason? Could this be
an Windows 2003 problem? Network configuration? We're not running a
firewall on our machine (don't tell anyone!), what else could be
interrupting the closing handshake? And why would this problem appear
all of a sudden if we haven't seen a significant increase in traffic
or we haven't changed our Apache configuration until we started seeing
problems?


Thanks!
Travis

On Sat, 2 Oct 2004 18:18:42 -0700, Gary Smith <gary@primeexalia.com> wrote:
> Travis,
> 
> What's the count of established connections?  They wouldn't by change be
> coming from the same IP holding your connection open would they?  The
> time_wait is waiting to receive a response back I think.  It's been a
> while since I dug through TCP stuff.  Anyways, you can have a big
> backlog queue without problem.  But if there are active connections and
> they are processing slowly (or hanging) then until they timeout Apache
> will continue to try to server those few.  Increasing the max
> connections in this case will have no effect.
> 
> 
> 
> 
> > -----Original Message-----
> > From: Travis Derouin [mailto:tderouin@gmail.com]
> > Sent: Saturday, October 02, 2004 3:57 PM
> > To: users@httpd.apache.org
> > Subject: [users@httpd] Apache 2.0.52 - stops responding
> >
> > Hi,
> >
> > I'm running Apache 2.0.52, with resin 2.1.14 on a Windows 2003 server.
> > The past few days we've been running into a weird problem, I'm not
> > sure whether or not Apache is responsible. Every few hours (our site
> > gets about 70k visitors a day), all the Apache worker threads get used
> > up and the server comes to halt. Doing a netstat -an, there are a lot
> > of sockets that show up as being either in the CLOSE_WAIT, TIME_WAIT,
> > or FIN_WAIT2 state.
> >
> > Example:
> >
> >   TCP    69.20.122.206:80       165.155.160.139:62360  FIN_WAIT_2
> >   TCP    69.20.122.206:80       167.80.246.204:10384   TIME_WAIT
> >   TCP    69.20.122.206:80       167.80.246.204:10392   TIME_WAIT
> >   TCP    69.20.122.206:80       167.80.246.204:10401   TIME_WAIT
> >   TCP    69.20.122.206:80       24.69.255.205:9963     ESTABLISHED
> >   TCP    69.20.122.206:80       24.77.151.229:1743     CLOSE_WAIT
> >   TCP    69.20.122.206:80       24.78.22.115:4466      CLOSE_WAIT
> >
> > When the process halts, of the 700 or so open connections to port 80,
> > 500 of them are in one of these 3 states (400 are CLOSE_WAIT).
> >
> > It's weird, everything was up and running for almost 3 months without
> > any problems, then since 9.29, apache comes to a complete halt every
> > few hours, depending on the traffic. We haven't run a windows update
> > since 9.15, so I doubt there's been anything new put on the machine
> > that's interefering with Apache. The worker threads seem to get used
> > up almost immediately before there's a crash, things will seem stable
> > according to the server status, with about 200 threads in use, then
> > it'll jump by a hundred per minute over the next few minutes until it
> > hits its max (at 1400) then everything stops working.
> >
> > Could the hosting provider have changed some network configuration
> > that might interfere with our Apache process? We also had the same
> > issues with 2.0.49, and I upgraded to .52. We haven't made any
> > significant code changes in the past month that might be responsible
> > for something like this.
> >
> > Any input would be appreciated! Thanks!!
> > Travis
> > 
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> >    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> 
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message