httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 44846] New: High load accumulates CLOSE_WAITs until server is unusable under Event MPM
Date Mon, 21 Apr 2008 15:27:55 GMT
https://issues.apache.org/bugzilla/show_bug.cgi?id=44846

           Summary: High load accumulates CLOSE_WAITs until server is
                    unusable under Event MPM
           Product: Apache httpd-2
           Version: 2.2.8
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: Event MPM
        AssignedTo: bugs@httpd.apache.org
        ReportedBy: justinbeech@gmail.com


While testing event MPM, using ab on another machine, if I pick
-c NNN (concurrency) greater than MaxClients, then although the
benchmark finishes reasonably, dangling CLOSE_WAIT connections
are left with Recv-Q data pointing to the httpd parent process.

Repeating the same highly concurrent ab a few more times causes
the CLOSE_WAIT connections to accumulate further until eventually
there are so many that httpd is no longer reachable at all.

When reaches this state, even after many minutes (or longer?)
it remains unreachable.. it must be killed (which ages the
CLOSE_WAITS very quickly) and restarted.

With lower concurrency the decay in performance to unreachability
is much longer but still present, and CLOSE_WAITs still 
accumulate roughly in proportion to how slow the server gets.

<IfModule event.c>
ListenBackLog      32
StartServers        10
ThreadLimit        50
ThreadsPerChild    50
MinSpareThreads    100
MaxSpareThreads    1000
ThreadsPerChild    50
MaxClients         500
MaxRequestsPerChild  0
</IfModule> 

KeepAlive is off.

Initial ab:

/ab -c 1000 -n 5000 http://192.168.1.1/server-status
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.1 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Finished 5000 requests


Server Software:        Apache
Server Hostname:        192.168.1.1
Server Port:            80

Document Path:          /server-status
Document Length:        215 bytes

Concurrency Level:      1000
Time taken for tests:   0.977488 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Non-2xx responses:      5014
Total transferred:      1900306 bytes
HTML transferred:       1078010 bytes
Requests per second:    5115.15 [#/sec] (mean)
Time per request:       195.498 [ms] (mean)
Time per request:       0.195 [ms] (mean, across all concurrent requests)
Transfer rate:          1897.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   15  10.7     13      41
Processing:     4   67 122.3     21     684
Waiting:        2   64 122.5     18     682
Total:          6   83 127.4     34     723

Percentage of the requests served within a certain time (ms)
  50%     34
  66%     45
  75%     65
  80%     83
  90%    257
  95%    279
  98%    676
  99%    680
 100%    723 (longest request)

Subsequent abs get lower and lower RPS ..

./ab -c 1000 -n 10000 http://192.168.1.1/
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests


Server Software:        Apache
Server Hostname:        192.168.1.1
Server Port:            80

Document Path:          /front/xxx
Document Length:        0 bytes

Concurrency Level:      1000
Time taken for tests:   17.232351 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      2370474 bytes
HTML transferred:       0 bytes
Requests per second:    580.30 [#/sec] (mean)
Time per request:       1723.235 [ms] (mean)
Time per request:       1.723 [ms] (mean, across all concurrent requests)
Transfer rate:          134.28 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   84 745.6      1    9023
Processing:    13  162 830.0     24   14539
Waiting:       11  160 830.0     22   14538
Total:         14  246 1150.7     25   16639

Percentage of the requests served within a certain time (ms)
  50%     25
  66%     30
  75%     41
  80%     52
  90%    140
  95%    669
  98%   3271
  99%   6669
 100%  16639 (longest request)

Until telnet ab or lynx just hang on the port.

And from netstat:

netstat -nap|grep CLOSE_WAIT | grep 192.168.1.1:80| more
tcp       96      0 192.168.1.1:80          192.168.1.30:49689      CLOSE_WAIT 
-                   
tcp       96      0 192.168.1.1:80          192.168.1.30:57625      CLOSE_WAIT 
-                   
tcp       96      0 192.168.1.1:80          192.168.1.30:37144      CLOSE_WAIT 
31414/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:57111      CLOSE_WAIT 
-                   
tcp       96      0 192.168.1.1:80          192.168.1.30:37143      CLOSE_WAIT 
31414/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:57110      CLOSE_WAIT 
-                   
tcp       96      0 192.168.1.1:80          192.168.1.30:37142      CLOSE_WAIT 
31414/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:51989      CLOSE_WAIT 
-                   
tcp       96      0 192.168.1.1:80          192.168.1.30:37141      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37141      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:51988      CLOSE_WAIT 
-                   
tcp       96      0 192.168.1.1:80          192.168.1.30:37140      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37138      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37137      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37138      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37137      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37136      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37135      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37134      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37133      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37132      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:54283      CLOSE_WAIT 
31469/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37131      CLOSE_WAIT 
31532/httpd         
tcp       96      0 192.168.1.1:80          192.168.1.30:37130      CLOSE_WAIT 
31532/httpd         

(etc)

Notice there are even some older entries from past runs where the server was
killed and restarted..

the kernel is 2.6.11 for this test, using epoll


-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message