Return-Path: Delivered-To: apache-bugdb-archive@hyperreal.org Received: (qmail 6767 invoked by uid 6000); 12 Oct 1998 15:00:12 -0000 Received: (qmail 6371 invoked by uid 2001); 12 Oct 1998 15:00:02 -0000 Received: (qmail 4520 invoked by uid 2012); 12 Oct 1998 14:55:43 -0000 Message-Id: <19981012145543.4519.qmail@hyperreal.org> Date: 12 Oct 1998 14:55:43 -0000 From: Kevin Martin Reply-To: sigma@pair.com To: apbugs@hyperreal.org X-Send-Pr-Version: 3.2 Subject: general/3190: httpd processes hang and never free up their slot, until MaxClients reached Sender: apache-bugdb-owner@apache.org Precedence: bulk >Number: 3190 >Category: general >Synopsis: httpd processes hang and never free up their slot, until MaxClients reached >Confidential: no >Severity: serious >Priority: medium >Responsible: apache >State: open >Class: sw-bug >Submitter-Id: apache >Arrival-Date: Mon Oct 12 08:00:01 PDT 1998 >Last-Modified: >Originator: sigma@pair.com >Organization: apache >Release: 1.2.6 >Environment: FreeBSD noldo.pair.com 2.1.7.1-RELEASE FreeBSD 2.1.7.1-RELEASE #1: Wed Nov 12 09:53:51 EST 1997 sigma@idad.pair.com:/usr/src/sys/compile/PAIRd i386 gcc version 2.6.3 (also observed under FreeBSD 2.2.7-STABLE) >Description: We have only observed this recently on two or three of our 150+ Web servers. What is happening is that on an otherwise normal and relatively quiet server, the httpd processes will stick around in the "R" state, with no activity, until eventually all slots are full. The most obvious datum is that on the mod_status report, the field Acc is always of the form X/Y/Y, where the last two numbers are the same. This is *not* true on any of our servers except for a couple. We have checked for SYN floods, unusual network activity, tried disabling busier sites on the server one at a time, cleaned up all .htaccess files, etc. We are not using serialized accept. We have tried with and without KeepAlive, at different levels. Our Timeout is set to 2400. I have verified that the configuration is identical to all other servers, except for the virtual hosts listed. ktrace on the parent shows only a series of sleeps and signals. ktrace on any child shows no system calls whatsoever. Attached below is the leftmost columns of our mod_status output after the server runs for a few minutes. Apache Status

Apache Server Status for *.pair.com

Current Time: Mon Oct 12 10:36:08 1998
Restart Time: Mon Oct 12 10:26:32 1998
Server uptime: 9 minutes 36 seconds
Total accesses: 3132 - Total Traffic: 12.0 MB
CPU Usage: u3.79688 s5.71875 cu14.6406 cs7.21094 - 5.45% CPU load
5.44 requests/sec - 21.4 kB/second - 4032 B/request
172 requests currently being processed, 17 idle servers

RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRKRRRRRRR_RRR_RRRRRRR___R____RR__RRR__RRR_WW__W_W...
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process

Srv PID Acc M CPU SS Conn Child Slot Host VHost Request

0 10628 0/20/20 R 0.42 458 0.0 0.14 0.14 199.227.99.68

1 10627 0/23/23 R 0.12 480 0.0 0.16 0.16 206.181.40.100

2 10626 0/0/0 R 0.00 572 0.0 0.00 0.00

3 10625 0/19/19 R 0.19 499 0.0 0.12 0.12 198.8.253.11

4 10624 0/5/5 R 0.02 556 0.0 0.02 0.02 193.14.10.129

5 10623 0/5/5 R 0.06 554 0.0 0.00 0.00 209.143.18.64

6 10622 0/0/0 R 0.00 572 0.0 0.00 0.00

7 10621 0/39/39 R 0.20 422 0.0 0.15 0.15 12.75.167.161

8 10620 0/21/21 R 0.05 518 0.0 0.07 0.07 147.155.3.52

9 10619 0/4/4 R 0.14 516 0.0 0.04 0.04 209.43.151.208

10 10618 0/15/15 R 0.34 498 0.0 0.06 0.06 207.226.104.4

11 10617 0/1/1 R 0.00 570 0.0 0.00 0.00 38.231.95.149

12 10616 0/42/42 R 0.35 402 0.0 0.13 0.13 152.163.217.99

13 10615 0/10/10 R 0.02 518 0.0 0.07 0.07 207.203.199.130

14 10614 0/8/8 R 0.08 537 0.0 0.05 0.05 167.240.254.1

15 10613 0/1/1 R 0.00 568 0.0 0.00 0.00 199.227.99.68

16 10612 0/6/6 R 0.02 553 0.0 0.03 0.03 143.183.152.10

17 10611 0/11/11 R 0.06 537 0.0 0.04 0.04 167.240.254.1

18 10610 0/0/0 R 0.00 571 0.0 0.00 0.00

19 10609 0/30/30 R 0.07 461 0.0 0.05 0.05 150.176.210.3

20 10608 0/26/26 R 0.27 480 0.0 0.14 0.14 206.181.40.100

21 10607 0/5/5 R 0.19 556 0.0 0.05 0.05 130.126.241.232

22 10606 0/20/20 R 0.08 479 0.0 0.11 0.11 209.48.216.1

23 10605 0/11/11 R 0.05 536 0.0 0.01 0.01 166.84.249.234

24 10637 0/16/16 R 0.17 517 0.0 0.04 0.04 167.240.254.1

25 10641 0/10/10 R 0.04 537 0.0 0.04 0.04 12.79.33.118

26 10643 0/6/6 R 0.02 553 0.0 0.02 0.02 143.183.152.10

27 10644 0/4/4 R 0.03 557 0.0 0.01 0.01 208.251.185.200

28 10650 0/97/97 R 1.05 79 0.0 0.62 0.62 199.100.49.105

29 10654 0/14/14 R 0.33 498 0.0 0.15 0.15 198.8.253.11

30 10665 0/25/25 R 0.25 458 0.0 0.05 0.05 198.8.253.11

31 10668 0/31/31 R 0.25 496 0.0 0.07 0.07 205.197.145.49

32 10671 0/69/69 R 0.88 288 0.0 0.26 0.26 209.99.57.63

33 10676 0/6/6 R 0.10 533 0.0 0.01 0.01 24.222.8.13

34 10680 0/28/28 R 0.20 324 0.0 0.02 0.02 206.186.80.120

35 10684 0/46/46 R 0.66 362 0.0 0.20 0.20 209.156.154.196

36 10685 0/11/11 R 0.03 460 0.0 0.11 0.11 195.180.246.99

37 10689 0/4/4 R 0.02 536 0.0 0.01 0.01 193.159.7.87

38 10717 0/28/28 R 0.35 440 0.0 0.09 0.09 205.197.145.49

39 10730 0/37/37 R 0.49 384 0.0 0.10 0.10 152.205.207.6

40 10734 0/0/0 R 0.00 530 0.0 0.00 0.00

41 10736 0/5/5 R 0.01 518 0.0 0.01 0.01 195.132.17.131

42 10739 0/16/16 R 0.23 459 0.0 0.07 0.07 210.225.125.39

43 10740 0/24/24 R 0.15 439 0.0 0.05 0.05 198.41.2.213

44 10745 0/2/2 R 0.18 515 0.0 0.00 0.00 207.24.17.3

45 10748 0/3/3 R 0.01 499 0.0 0.04 0.04 208.251.185.48

46 10752 0/13/13 R 0.24 440 0.0 0.07 0.07 198.41.2.213

47 10754 0/5/5 R 0.07 499 0.0 0.00 0.00 209.43.151.208

48 10760 0/8/8 R 0.09 480 0.0 0.02 0.02 206.181.40.100

49 10768 0/8/8 R 0.23 477 0.0 0.02 0.02 209.48.216.1

50 10804 0/3/3 R 0.15 461 0.0 0.04 0.04 157.193.124.79

51 10806 0/5/5 R 0.01 479 0.0 0.00 0.00 206.156.188.195

52 10810 0/40/40 R 0.27 342 0.0 0.13 0.13 12.2.40.50

53 10835 0/11/11 R 0.25 423 0.0 0.03 0.03 130.244.106.149

54 10842 0/4/4 R 0.02 440 0.0 0.01 0.01 198.41.2.213

55 10844 0/72/72 R 0.47 269 0.0 0.15 0.15 206.230.56.98

56 10845 0/7/7 R 0.17 422 0.0 0.06 0.06 12.75.167.161

57 10847 0/25/25 R 0.27 364 0.0 0.24 0.24 192.128.133.43

58 10850 0/15/15 R 0.04 403 0.0 0.01 0.01 205.188.154.131

59 10854 0/4/4 R 0.15 439 0.0 0.03 0.03 206.156.170.138

60 10856 0/32/32 R 0.36 289 0.0 0.07 0.07 206.156.170.138

61 10868 0/6/6 R 0.08 420 0.0 0.00 0.00 195.7.226.214

62 10894 0/34/34 R 0.13 345 0.0 0.09 0.09 12.2.40.50

63 10896 0/8/8 R 0.17 401 0.0 0.06 0.06 193.237.223.56

64 10898 0/3/3 R 0.01 421 0.0 0.00 0.00 198.51.197.28

65 10900 0/1/1 R 0.01 426 0.0 0.00 0.00 203.18.138.18

66 10909 0/13/13 R 0.20 383 0.0 0.06 0.06 152.175.235.63

67 10911 0/0/0 R 0.00 423 0.0 0.00 0.00

68 10916 0/2/2 R 0.00 402 0.0 0.04 0.04 152.205.207.6

69 10918 0/67/67 R 0.38 288 0.0 0.14 0.14 193.237.223.56

70 10919 0/3/3 R 0.01 403 0.0 0.00 0.00 205.188.154.132

71 10921 0/2/2 R 0.01 404 0.0 0.00 0.00 169.204.27.50

72 10932 0/21/21 R 0.23 345 0.0 0.03 0.03 208.255.231.175

73 10934 0/14/14 R 0.05 364 0.0 0.03 0.03 128.173.212.60

74 10939 0/7/7 R 0.01 382 0.0 0.03 0.03 129.206.42.13

75 10940 0/25/25 R 0.08 308 0.0 0.06 0.06 156.46.120.186

76 10942 0/26/26 R 0.20 307 0.0 0.02 0.02 198.66.160.106

77 10944 0/38/38 R 0.15 248 0.0 0.09 0.09 128.253.67.102

78 10946 0/3/3 R 0.04 385 0.0 0.01 0.01 141.52.28.13

79 10950 0/10/10 R 0.05 364 0.0 0.03 0.03 132.192.4.148

80 10951 0/15/15 R 0.29 346 0.0 0.03 0.03 208.255.231.175

81 10961 0/22/22 R 0.19 327 0.0 0.07 0.07 153.37.128.92

82 10970 0/0/0 R 0.00 383 0.0 0.00 0.00

83 10973 0/33/33 R 0.22 269 0.0 0.09 0.09 208.6.199.41

84 10974 0/0/0 R 0.00 381 0.0 0.00 0.00

85 10975 0/65/65 R 0.52 152 0.0 0.26 0.26 129.2.103.26

86 10977 0/45/45 R 0.48 177 0.0 0.16 0.16 193.213.53.1

87 10979 0/6/6 R 0.15 327 0.0 0.07 0.07 195.10.0.198

88 10987 0/3/3 R 0.03 364 0.0 0.00 0.00 194.78.125.242

89 10989 0/3/3 R 0.00 365 0.0 0.00 0.00 195.7.226.214

90 10994 0/20/20 R 0.24 270 0.0 0.05 0.05 209.54.108.139

91 10997 0/18/18 R 0.84 289 0.0 0.05 0.05 206.156.170.138

92 10999 0/13/13 R 0.02 308 0.0 0.03 0.03 198.66.160.106

93 11003 0/56/56 R 0.39 173 0.0 0.22 0.22 132.235.47.11

94 11005 0/10/10 R 0.16 326 0.0 0.08 0.08 198.247.68.197

95 11014 0/11/11 R 0.11 307 0.0 0.05 0.05 194.240.179.25

96 11017 0/4/4 R 0.01 346 0.0 0.00 0.00 206.186.80.120

97 11020 0/3/3 R 0.00 344 0.0 0.00 0.00 210.111.158.83

98 11031 0/53/53 R 0.32 133 0.0 0.11 0.11 194.109.6.90

99 11032 0/4/4 R 0.02 326 0.0 0.03 0.03 207.2.199.229

100 11034 0/9/9 R 0.05 327 0.0 0.04 0.04 131.220.65.91

101 11045 0/32/32 R 0.55 251 0.0 0.10 0.10 128.253.67.102

102 11046 0/4/4 R 0.02 318 0.0 0.06 0.06 161.57.50.35

103 11048 0/21/21 R 0.13 229 0.0 0.05 0.05 153.37.114.96

104 11050 0/9/9 R 0.17 286 0.0 0.11 0.11 206.156.170.138

105 11051 0/37/37 R 0.27 231 0.0 0.10 0.10 208.212.66.66

106 11060 0/18/18 R 0.05 270 0.0 0.00 0.00 205.209.21.126

107 11089 0/3/3 R 0.16 304 0.0 0.03 0.03 206.186.80.120

108 11102 0/9/9 R 0.02 288 0.0 0.00 0.00 206.140.85.97

109 11103 0/1/1 R 0.00 308 0.0 0.00 0.00 194.152.3.181

110 11130 0/25/25 R 0.19 230 0.0 0.01 0.01 208.139.201.117

111 11138 0/7/7 R 0.34 251 0.0 0.05 0.05 153.36.36.169

112 11140 0/55/55 R 0.60 119 0.0 0.30 0.30 209.240.198.181

113 11142 0/5/5 R 0.04 267 0.0 0.02 0.02 192.89.216.2

114 11143 0/4/4 R 0.01 269 0.0 0.00 0.00 209.54.108.139

115 11145 0/18/18 R 0.16 115 0.0 0.03 0.03 141.214.182.250

116 11148 0/33/33 R 0.33 175 0.0 0.07 0.07 137.186.209.37

117 11155 0/12/12 R 0.04 231 0.0 0.03 0.03 130.38.163.247

118 11159 0/5/5 R 0.07 172 0.0 0.00 0.00 12.3.14.51

119 11161 0/60/60 R 0.44 78 0.0 0.16 0.16 206.53.113.239

120 11177 0/26/26 R 0.09 154 0.0 0.12 0.12 212.211.28.19

121 11180 0/4/4 R 0.02 250 0.0 0.00 0.00 152.204.32.218

122 11185 0/25/25 R 0.58 173 0.0 0.14 0.14 12.3.14.51

123 11191 0/26/26 R 0.27 171 0.0 0.17 0.17 209.68.1.26

124 11196 0/3/3 R 0.02 248 0.0 0.00 0.00 128.253.67.102

125 11207 0/1/1 R 0.01 250 0.0 0.00 0.00 207.59.113.196

126 11208 0/40/40 R 0.27 117 0.0 0.11 0.11 24.128.92.239

127 11214 0/7/7 R 0.08 232 0.0 0.01 0.01 207.173.246.164

128 11220 0/20/20 R 0.12 156 0.0 0.08 0.08 134.96.100.107

129 11228 0/21/21 R 0.20 177 0.0 0.03 0.03 137.186.209.37

130 11229 0/3/3 R 0.01 230 0.0 0.01 0.01 199.238.253.15

131 11232 0/24/24 R 0.68 176 0.0 0.11 0.11 198.66.160.106

132 11253 0/21/21 R 0.02 174 0.0 0.00 0.00 141.163.29.76

133 11255 0/39/39 R 0.83 74 0.0 0.36 0.36 206.53.113.239

134 11262 0/18/18 R 0.20 177 0.0 0.02 0.02 193.213.53.1

135 11267 0/5/5 R 0.05 173 0.0 0.05 0.05 129.2.103.26

136 11280 0/20/20 R 0.24 136 0.0 0.09 0.09 129.137.68.75

137 11293 0/9/9 R 0.02 173 0.0 0.05 0.05 12.3.14.51

138 11296 0/13/13 R 0.21 133 0.0 0.08 0.08 153.37.114.96

139 11298 0/5/5 R 0.04 178 0.0 0.02 0.02 194.109.6.90

140 11299 0/3/3 R 0.01 154 0.0 0.02 0.02 159.134.226.244

141 11302 1/46/46 K 0.58 2 0.6 0.17 0.17 209.167.3.194

142 11304 0/7/7 R 0.02 174 0.0 0.02 0.02 132.235.47.11

143 11320 0/3/3 R 0.01 176 0.0 0.00 0.00 212.216.99.7

144 11330 0/0/0 R 0.00 176 0.0 0.00 0.00

145 11338 0/4/4 R 0.01 150 0.0 0.02 0.02 159.134.226.244

146 11339 0/4/4 R 0.16 155 0.0 0.04 0.04 208.212.66.66

147 11340 0/21/21 R 0.15 97 0.0 0.08 0.08 159.116.252.3

148 11344 0/17/17 R 0.17 101 0.0 0.03 0.03 210.111.158.83

149 11349 0/39/39 _ 0.25 4 0.0 0.17 0.17 209.1.12.51

150 11352 0/13/13 R 0.37 116 0.0 0.07 0.07 150.198.244.29

151 11354 0/6/6 R 0.02 136 0.0 0.01 0.01 159.134.226.244

152 11378 0/40/40 R 0.38 1 0.0 0.18 0.18 198.59.148.11

153 11380 0/56/56 _ 0.54 2 0.0 0.22 0.22 129.132.31.7

154 11388 0/3/3 R 0.02 136 0.0 0.00 0.00 129.137.68.75

155 11395 0/9/9 R 0.04 98 0.0 0.01 0.01 204.128.192.59

156 11397 0/11/11 R 0.05 97 0.0 0.02 0.02 159.116.252.3

157 11401 0/0/0 R 0.00 138 0.0 0.00 0.00

158 11402 0/13/13 R 0.05 61 0.0 0.13 0.13 194.95.213.113

159 11417 0/3/3 R 0.02 118 0.0 0.00 0.00 209.240.198.186

160 11419 0/12/12 R 0.01 98 0.0 0.02 0.02 207.94.105.103

161 11459 0/30/30 _ 0.35 3 0.0 0.11 0.11 208.240.114.202

162 11476 0/19/19 _ 0.21 7 0.0 0.16 0.16 198.59.148.11

163 11478 0/9/9 _ 0.05 0 0.0 0.04 0.04 129.71.64.157

164 11479 0/6/6 R 0.17 79 0.0 0.02 0.02 203.56.239.200

165 11482 0/34/34 _ 0.45 4 0.0 0.13 0.13 129.132.31.7

166 11483 0/39/39 _ 0.27 5 0.0 0.11 0.11 208.143.10.120

167 11484 0/28/28 _ 0.20 6 0.0 0.27 0.27 205.188.154.1

168 11498 0/24/24 _ 0.22 1 0.0 0.12 0.12 129.132.31.7

169 11501 0/1/1 R 0.01 96 0.0 0.00 0.00 159.116.252.3

170 11503 0/4/4 R 0.02 80 0.0 0.06 0.06 199.100.49.105

171 11505 0/24/24 _ 0.23 2 0.0 0.13 0.13 193.158.139.113

172 11517 0/23/23 _ 0.08 6 0.0 0.11 0.11 129.132.31.7

173 11521 0/3/3 R 0.01 78 0.0 0.00 0.00 204.178.77.46

174 11528 0/1/1 R 0.00 75 0.0 0.00 0.00 62.112.0.10

175 11538 0/4/4 R 0.01 61 0.0 0.00 0.00 207.13.93.102

177 11543 0/22/22 _ 0.12 6 0.0 0.04 0.04 194.109.6.90

178 11547 0/12/12 R 0.05 19 0.0 0.04 0.04 207.220.84.100

179 11549 0/20/20 R 0.06 0 0.0 0.06 0.06 129.71.64.157

180 11565 1/17/17 W 0.45 0 0.0 0.07 0.07 196.28.7.31

181 11569 0/16/16 _ 0.27 4 0.0 0.05 0.05 129.132.31.7

182 11573 0/11/11 W 0.41 0 0.0 0.11 0.11 209.68.1.26

183 11579 0/5/5 W 0.04 28 0.0 0.03 0.03 207.172.166.206

184 11582 0/16/16 _ 0.03 0 0.0 0.11 0.11 129.71.64.157

185 11584 0/12/12 _ 0.05 7 0.0 0.07 0.07 129.132.31.7

186 11595 1/12/12 W 0.38 1 0.0 0.06 0.06 194.133.33.19

187 11634 0/5/5 _ 0.07 4 0.0 0.00 0.00 129.71.64.157

188 11636 1/6/6 W 0.02 0 0.0 0.01 0.01 194.18.163.1

Srv	PID	Acc	M	CPU	SS	Conn	Child	Slot	Host	VHost	Request
0	10628	0/20/20	R	0.42	458	0.0	0.14	0.14	199.227.99.68
1	10627	0/23/23	R	0.12	480	0.0	0.16	0.16	206.181.40.100
2	10626	0/0/0	R	0.00	572	0.0	0.00	0.00
3	10625	0/19/19	R	0.19	499	0.0	0.12	0.12	198.8.253.11
4	10624	0/5/5	R	0.02	556	0.0	0.02	0.02	193.14.10.129
5	10623	0/5/5	R	0.06	554	0.0	0.00	0.00	209.143.18.64
6	10622	0/0/0	R	0.00	572	0.0	0.00	0.00
7	10621	0/39/39	R	0.20	422	0.0	0.15	0.15	12.75.167.161
8	10620	0/21/21	R	0.05	518	0.0	0.07	0.07	147.155.3.52
9	10619	0/4/4	R	0.14	516	0.0	0.04	0.04	209.43.151.208
10	10618	0/15/15	R	0.34	498	0.0	0.06	0.06	207.226.104.4
11	10617	0/1/1	R	0.00	570	0.0	0.00	0.00	38.231.95.149
12	10616	0/42/42	R	0.35	402	0.0	0.13	0.13	152.163.217.99
13	10615	0/10/10	R	0.02	518	0.0	0.07	0.07	207.203.199.130
14	10614	0/8/8	R	0.08	537	0.0	0.05	0.05	167.240.254.1
15	10613	0/1/1	R	0.00	568	0.0	0.00	0.00	199.227.99.68
16	10612	0/6/6	R	0.02	553	0.0	0.03	0.03	143.183.152.10
17	10611	0/11/11	R	0.06	537	0.0	0.04	0.04	167.240.254.1
18	10610	0/0/0	R	0.00	571	0.0	0.00	0.00
19	10609	0/30/30	R	0.07	461	0.0	0.05	0.05	150.176.210.3
20	10608	0/26/26	R	0.27	480	0.0	0.14	0.14	206.181.40.100
21	10607	0/5/5	R	0.19	556	0.0	0.05	0.05	130.126.241.232
22	10606	0/20/20	R	0.08	479	0.0	0.11	0.11	209.48.216.1
23	10605	0/11/11	R	0.05	536	0.0	0.01	0.01	166.84.249.234
24	10637	0/16/16	R	0.17	517	0.0	0.04	0.04	167.240.254.1
25	10641	0/10/10	R	0.04	537	0.0	0.04	0.04	12.79.33.118
26	10643	0/6/6	R	0.02	553	0.0	0.02	0.02	143.183.152.10
27	10644	0/4/4	R	0.03	557	0.0	0.01	0.01	208.251.185.200
28	10650	0/97/97	R	1.05	79	0.0	0.62	0.62	199.100.49.105
29	10654	0/14/14	R	0.33	498	0.0	0.15	0.15	198.8.253.11
30	10665	0/25/25	R	0.25	458	0.0	0.05	0.05	198.8.253.11
31	10668	0/31/31	R	0.25	496	0.0	0.07	0.07	205.197.145.49
32	10671	0/69/69	R	0.88	288	0.0	0.26	0.26	209.99.57.63
33	10676	0/6/6	R	0.10	533	0.0	0.01	0.01	24.222.8.13
34	10680	0/28/28	R	0.20	324	0.0	0.02	0.02	206.186.80.120
35	10684	0/46/46	R	0.66	362	0.0	0.20	0.20	209.156.154.196
36	10685	0/11/11	R	0.03	460	0.0	0.11	0.11	195.180.246.99
37	10689	0/4/4	R	0.02	536	0.0	0.01	0.01	193.159.7.87
38	10717	0/28/28	R	0.35	440	0.0	0.09	0.09	205.197.145.49
39	10730	0/37/37	R	0.49	384	0.0	0.10	0.10	152.205.207.6
40	10734	0/0/0	R	0.00	530	0.0	0.00	0.00
41	10736	0/5/5	R	0.01	518	0.0	0.01	0.01	195.132.17.131
42	10739	0/16/16	R	0.23	459	0.0	0.07	0.07	210.225.125.39
43	10740	0/24/24	R	0.15	439	0.0	0.05	0.05	198.41.2.213
44	10745	0/2/2	R	0.18	515	0.0	0.00	0.00	207.24.17.3
45	10748	0/3/3	R	0.01	499	0.0	0.04	0.04	208.251.185.48
46	10752	0/13/13	R	0.24	440	0.0	0.07	0.07	198.41.2.213
47	10754	0/5/5	R	0.07	499	0.0	0.00	0.00	209.43.151.208
48	10760	0/8/8	R	0.09	480	0.0	0.02	0.02	206.181.40.100
49	10768	0/8/8	R	0.23	477	0.0	0.02	0.02	209.48.216.1
50	10804	0/3/3	R	0.15	461	0.0	0.04	0.04	157.193.124.79
51	10806	0/5/5	R	0.01	479	0.0	0.00	0.00	206.156.188.195
52	10810	0/40/40	R	0.27	342	0.0	0.13	0.13	12.2.40.50
53	10835	0/11/11	R	0.25	423	0.0	0.03	0.03	130.244.106.149
54	10842	0/4/4	R	0.02	440	0.0	0.01	0.01	198.41.2.213
55	10844	0/72/72	R	0.47	269	0.0	0.15	0.15	206.230.56.98
56	10845	0/7/7	R	0.17	422	0.0	0.06	0.06	12.75.167.161
57	10847	0/25/25	R	0.27	364	0.0	0.24	0.24	192.128.133.43
58	10850	0/15/15	R	0.04	403	0.0	0.01	0.01	205.188.154.131
59	10854	0/4/4	R	0.15	439	0.0	0.03	0.03	206.156.170.138
60	10856	0/32/32	R	0.36	289	0.0	0.07	0.07	206.156.170.138
61	10868	0/6/6	R	0.08	420	0.0	0.00	0.00	195.7.226.214
62	10894	0/34/34	R	0.13	345	0.0	0.09	0.09	12.2.40.50
63	10896	0/8/8	R	0.17	401	0.0	0.06	0.06	193.237.223.56
64	10898	0/3/3	R	0.01	421	0.0	0.00	0.00	198.51.197.28
65	10900	0/1/1	R	0.01	426	0.0	0.00	0.00	203.18.138.18
66	10909	0/13/13	R	0.20	383	0.0	0.06	0.06	152.175.235.63
67	10911	0/0/0	R	0.00	423	0.0	0.00	0.00
68	10916	0/2/2	R	0.00	402	0.0	0.04	0.04	152.205.207.6
69	10918	0/67/67	R	0.38	288	0.0	0.14	0.14	193.237.223.56
70	10919	0/3/3	R	0.01	403	0.0	0.00	0.00	205.188.154.132
71	10921	0/2/2	R	0.01	404	0.0	0.00	0.00	169.204.27.50
72	10932	0/21/21	R	0.23	345	0.0	0.03	0.03	208.255.231.175
73	10934	0/14/14	R	0.05	364	0.0	0.03	0.03	128.173.212.60
74	10939	0/7/7	R	0.01	382	0.0	0.03	0.03	129.206.42.13
75	10940	0/25/25	R	0.08	308	0.0	0.06	0.06	156.46.120.186
76	10942	0/26/26	R	0.20	307	0.0	0.02	0.02	198.66.160.106
77	10944	0/38/38	R	0.15	248	0.0	0.09	0.09	128.253.67.102
78	10946	0/3/3	R	0.04	385	0.0	0.01	0.01	141.52.28.13
79	10950	0/10/10	R	0.05	364	0.0	0.03	0.03	132.192.4.148
80	10951	0/15/15	R	0.29	346	0.0	0.03	0.03	208.255.231.175
81	10961	0/22/22	R	0.19	327	0.0	0.07	0.07	153.37.128.92
82	10970	0/0/0	R	0.00	383	0.0	0.00	0.00
83	10973	0/33/33	R	0.22	269	0.0	0.09	0.09	208.6.199.41
84	10974	0/0/0	R	0.00	381	0.0	0.00	0.00
85	10975	0/65/65	R	0.52	152	0.0	0.26	0.26	129.2.103.26
86	10977	0/45/45	R	0.48	177	0.0	0.16	0.16	193.213.53.1
87	10979	0/6/6	R	0.15	327	0.0	0.07	0.07	195.10.0.198
88	10987	0/3/3	R	0.03	364	0.0	0.00	0.00	194.78.125.242
89	10989	0/3/3	R	0.00	365	0.0	0.00	0.00	195.7.226.214
90	10994	0/20/20	R	0.24	270	0.0	0.05	0.05	209.54.108.139
91	10997	0/18/18	R	0.84	289	0.0	0.05	0.05	206.156.170.138
92	10999	0/13/13	R	0.02	308	0.0	0.03	0.03	198.66.160.106
93	11003	0/56/56	R	0.39	173	0.0	0.22	0.22	132.235.47.11
94	11005	0/10/10	R	0.16	326	0.0	0.08	0.08	198.247.68.197
95	11014	0/11/11	R	0.11	307	0.0	0.05	0.05	194.240.179.25
96	11017	0/4/4	R	0.01	346	0.0	0.00	0.00	206.186.80.120
97	11020	0/3/3	R	0.00	344	0.0	0.00	0.00	210.111.158.83
98	11031	0/53/53	R	0.32	133	0.0	0.11	0.11	194.109.6.90
99	11032	0/4/4	R	0.02	326	0.0	0.03	0.03	207.2.199.229
100	11034	0/9/9	R	0.05	327	0.0	0.04	0.04	131.220.65.91
101	11045	0/32/32	R	0.55	251	0.0	0.10	0.10	128.253.67.102
102	11046	0/4/4	R	0.02	318	0.0	0.06	0.06	161.57.50.35
103	11048	0/21/21	R	0.13	229	0.0	0.05	0.05	153.37.114.96
104	11050	0/9/9	R	0.17	286	0.0	0.11	0.11	206.156.170.138
105	11051	0/37/37	R	0.27	231	0.0	0.10	0.10	208.212.66.66
106	11060	0/18/18	R	0.05	270	0.0	0.00	0.00	205.209.21.126
107	11089	0/3/3	R	0.16	304	0.0	0.03	0.03	206.186.80.120
108	11102	0/9/9	R	0.02	288	0.0	0.00	0.00	206.140.85.97
109	11103	0/1/1	R	0.00	308	0.0	0.00	0.00	194.152.3.181
110	11130	0/25/25	R	0.19	230	0.0	0.01	0.01	208.139.201.117
111	11138	0/7/7	R	0.34	251	0.0	0.05	0.05	153.36.36.169
112	11140	0/55/55	R	0.60	119	0.0	0.30	0.30	209.240.198.181
113	11142	0/5/5	R	0.04	267	0.0	0.02	0.02	192.89.216.2
114	11143	0/4/4	R	0.01	269	0.0	0.00	0.00	209.54.108.139
115	11145	0/18/18	R	0.16	115	0.0	0.03	0.03	141.214.182.250
116	11148	0/33/33	R	0.33	175	0.0	0.07	0.07	137.186.209.37
117	11155	0/12/12	R	0.04	231	0.0	0.03	0.03	130.38.163.247
118	11159	0/5/5	R	0.07	172	0.0	0.00	0.00	12.3.14.51
119	11161	0/60/60	R	0.44	78	0.0	0.16	0.16	206.53.113.239
120	11177	0/26/26	R	0.09	154	0.0	0.12	0.12	212.211.28.19
121	11180	0/4/4	R	0.02	250	0.0	0.00	0.00	152.204.32.218
122	11185	0/25/25	R	0.58	173	0.0	0.14	0.14	12.3.14.51
123	11191	0/26/26	R	0.27	171	0.0	0.17	0.17	209.68.1.26
124	11196	0/3/3	R	0.02	248	0.0	0.00	0.00	128.253.67.102
125	11207	0/1/1	R	0.01	250	0.0	0.00	0.00	207.59.113.196
126	11208	0/40/40	R	0.27	117	0.0	0.11	0.11	24.128.92.239
127	11214	0/7/7	R	0.08	232	0.0	0.01	0.01	207.173.246.164
128	11220	0/20/20	R	0.12	156	0.0	0.08	0.08	134.96.100.107
129	11228	0/21/21	R	0.20	177	0.0	0.03	0.03	137.186.209.37
130	11229	0/3/3	R	0.01	230	0.0	0.01	0.01	199.238.253.15
131	11232	0/24/24	R	0.68	176	0.0	0.11	0.11	198.66.160.106
132	11253	0/21/21	R	0.02	174	0.0	0.00	0.00	141.163.29.76
133	11255	0/39/39	R	0.83	74	0.0	0.36	0.36	206.53.113.239
134	11262	0/18/18	R	0.20	177	0.0	0.02	0.02	193.213.53.1
135	11267	0/5/5	R	0.05	173	0.0	0.05	0.05	129.2.103.26
136	11280	0/20/20	R	0.24	136	0.0	0.09	0.09	129.137.68.75
137	11293	0/9/9	R	0.02	173	0.0	0.05	0.05	12.3.14.51
138	11296	0/13/13	R	0.21	133	0.0	0.08	0.08	153.37.114.96
139	11298	0/5/5	R	0.04	178	0.0	0.02	0.02	194.109.6.90
140	11299	0/3/3	R	0.01	154	0.0	0.02	0.02	159.134.226.244
141	11302	1/46/46	K	0.58	2	0.6	0.17	0.17	209.167.3.194
142	11304	0/7/7	R	0.02	174	0.0	0.02	0.02	132.235.47.11
143	11320	0/3/3	R	0.01	176	0.0	0.00	0.00	212.216.99.7
144	11330	0/0/0	R	0.00	176	0.0	0.00	0.00
145	11338	0/4/4	R	0.01	150	0.0	0.02	0.02	159.134.226.244
146	11339	0/4/4	R	0.16	155	0.0	0.04	0.04	208.212.66.66
147	11340	0/21/21	R	0.15	97	0.0	0.08	0.08	159.116.252.3
148	11344	0/17/17	R	0.17	101	0.0	0.03	0.03	210.111.158.83
149	11349	0/39/39	_	0.25	4	0.0	0.17	0.17	209.1.12.51
150	11352	0/13/13	R	0.37	116	0.0	0.07	0.07	150.198.244.29
151	11354	0/6/6	R	0.02	136	0.0	0.01	0.01	159.134.226.244
152	11378	0/40/40	R	0.38	1	0.0	0.18	0.18	198.59.148.11
153	11380	0/56/56	_	0.54	2	0.0	0.22	0.22	129.132.31.7
154	11388	0/3/3	R	0.02	136	0.0	0.00	0.00	129.137.68.75
155	11395	0/9/9	R	0.04	98	0.0	0.01	0.01	204.128.192.59
156	11397	0/11/11	R	0.05	97	0.0	0.02	0.02	159.116.252.3
157	11401	0/0/0	R	0.00	138	0.0	0.00	0.00
158	11402	0/13/13	R	0.05	61	0.0	0.13	0.13	194.95.213.113
159	11417	0/3/3	R	0.02	118	0.0	0.00	0.00	209.240.198.186
160	11419	0/12/12	R	0.01	98	0.0	0.02	0.02	207.94.105.103
161	11459	0/30/30	_	0.35	3	0.0	0.11	0.11	208.240.114.202
162	11476	0/19/19	_	0.21	7	0.0	0.16	0.16	198.59.148.11
163	11478	0/9/9	_	0.05	0	0.0	0.04	0.04	129.71.64.157
164	11479	0/6/6	R	0.17	79	0.0	0.02	0.02	203.56.239.200
165	11482	0/34/34	_	0.45	4	0.0	0.13	0.13	129.132.31.7
166	11483	0/39/39	_	0.27	5	0.0	0.11	0.11	208.143.10.120
167	11484	0/28/28	_	0.20	6	0.0	0.27	0.27	205.188.154.1
168	11498	0/24/24	_	0.22	1	0.0	0.12	0.12	129.132.31.7
169	11501	0/1/1	R	0.01	96	0.0	0.00	0.00	159.116.252.3
170	11503	0/4/4	R	0.02	80	0.0	0.06	0.06	199.100.49.105
171	11505	0/24/24	_	0.23	2	0.0	0.13	0.13	193.158.139.113
172	11517	0/23/23	_	0.08	6	0.0	0.11	0.11	129.132.31.7
173	11521	0/3/3	R	0.01	78	0.0	0.00	0.00	204.178.77.46
174	11528	0/1/1	R	0.00	75	0.0	0.00	0.00	62.112.0.10
175	11538	0/4/4	R	0.01	61	0.0	0.00	0.00	207.13.93.102
177	11543	0/22/22	_	0.12	6	0.0	0.04	0.04	194.109.6.90
178	11547	0/12/12	R	0.05	19	0.0	0.04	0.04	207.220.84.100
179	11549	0/20/20	R	0.06	0	0.0	0.06	0.06	129.71.64.157
180	11565	1/17/17	W	0.45	0	0.0	0.07	0.07	196.28.7.31
181	11569	0/16/16	_	0.27	4	0.0	0.05	0.05	129.132.31.7
182	11573	0/11/11	W	0.41	0	0.0	0.11	0.11	209.68.1.26
183	11579	0/5/5	W	0.04	28	0.0	0.03	0.03	207.172.166.206
184	11582	0/16/16	_	0.03	0	0.0	0.11	0.11	129.71.64.157
185	11584	0/12/12	_	0.05	7	0.0	0.07	0.07	129.132.31.7
186	11595	1/12/12	W	0.38	1	0.0	0.06	0.06	194.133.33.19
187	11634	0/5/5	_	0.07	4	0.0	0.00	0.00	129.71.64.157
188	11636	1/6/6	W	0.02	0	0.0	0.01	0.01	194.18.163.1

Srv	Server number
PID	OS process ID
Acc	Number of accesses this connection / this child / this slot
M	Mode of operation
CPU	CPU usage, number of seconds
SS	Seconds since beginning of most recent request
Conn	Kilobytes transferred this connection
Child	Megabytes transferred this child
Slot	Total megabytes transferred this slot

>How-To-Repeat: Unknown. We once "fixed" a server when we found a .htaccess file that had no carriage return in it - but this does not appear to be the case in the most recent incidents. I believe this is the same problem described in the PR #1724, which was never completely understood and resolved. >Fix: Something is causing httpd processes not to return to the idle state. We are working on an upgrade to Apache 1.3.3 to see if this resolves the problem. >Audit-Trail: >Unformatted: [In order for any reply to be added to the PR database, ] [you need to include in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ]