httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Whitton <tinymount...@gmail.com>
Subject [users@httpd] Slow Responses Under Low Load
Date Tue, 31 Aug 2010 20:09:30 GMT
Hi there, we're having a strange issue and wondering if anyone might
have any suggestions. We have a cluster of apache machines and are
experiencing a problem where the load average and other hardware
metrics are well below normal ranges, but the response times are
exceedingly high (even for tiny pieces of static content transferred
over localhost). We've been running Apache in production for several
years now without any issues, and this problem just started yesterday.
This likely means the problem isn't directly caused by Apache but is
affecting it nonetheless. If we take an httpd server out of the pool
via our load balancer, the response times go back to normal. Here's an
example of the behavior I'm describing.

[root@RHL060 tmp]# time wget http://apc.rhl060.in.xxxxxxx.com/foo.html
--2010-08-31 15:49:43--  http://apc.rhl060.in.xxxxxxx.com/foo.html
Resolving apc.rhl060.in.xxxxxxx.com... 127.0.0.1
Connecting to apc.rhl060.in.xxxxxxx.com|127.0.0.1|:80... failed:
Connection timed out.
Retrying.

--2010-08-31 15:52:53--  (try: 2)  http://apc.rhl060.in.xxxxxxx.com/foo.html
Connecting to apc.rhl060.in.xxxxxxx.com|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4 [text/html]
Saving to: `foo.html.72'

2010-08-31 15:53:14 (558 KB/s) - `foo.html.72' saved [4/4]

real	3m31.004s
user	0m0.000s
sys	0m0.003s

For the problem illustrated above, the loopback has no iptables rules
applied. Running `netstat -tapn | grep 80 | wc -l' shows 11299
connections to port 80. We're setup to use prefork with the following
configuration.

<IfModule prefork.c>
StartServers       100
MinSpareServers    100
MaxSpareServers   200
ServerLimit       1000
MaxClients        1000
MaxRequestsPerChild  100000
</IfModule>

# uptime
 16:06:47 up 14 days, 17:47,  5 users,  load average: 2.45, 2.54, 2.74

Checking iostat shows very little disk utilization.

The server is running CentOS 5.5 with kernel version
2.6.18-194.8.1.el5. This particular server has 24 cores and 50G of
RAM. A number of our servers are experiencing the same issue, and they
all have similar hardware and software specs.

I'm pretty stumped here. Any help would be highly appreciated.

-Travis

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message