httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Roussey" <srous...@network54.com>
Subject Re: What is the source of this 'Thundering herd' redux?
Date Fri, 03 Jan 2003 03:23:04 GMT
> Can you grab a stack trace from each of the httpd processes (e.g.,
> with pstack, or by scripting gdb) to see what they're doing when
> the load drops to zero?

Yes, but... pstack does not find any symbols (gdb does). Any way around
this? Am I compiling apache incorrectly?

Also, I like this idea. I'll write a script to run and wait until the
idle time is over 60% and then start logging a pstack of all apache
processes. I have no clue how to do that, but I'll figure it out.

> Also, does netstat show connections piling up in 'established' 
> state on the port on which your httpd is listening?

Yes, even while idle is high. What does this mean???

Note that the load goes way down while the established connections rise
-- for several seconds. No processes are waiting for run time. Then,
bang! High load and hundreds of processes waiting for run time and then
lots waiting for run and lots in 'uninterruptable sleep'. It is madness!
Load goes from 5-15 to 60-220! 



Remember, connections here will include both those used by Apache to the
client (no Keep-Alive) and by Apache to the MySQL server (persistent is
yes). Max clients for Apache is 410, so the max connections should be
820-860 or so for Apache. (There is a named running using a few as
well).

while (echo 1); do vmstat 1 2; netstat --statistics | grep 'connections
est';done


   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 5  0  0 114300  97428   4672  49144  28  50    88    75  111   357  59
22  19
 8  0  0 114300  97436   4684  49180   0   0    40    16 10173  8251  46
17  38
    365 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
12  0  1 114300  97436   4684  49180  28  50    88    75  112   357  59
22  19
35  0  1 114300  97436   4704  49316   0   0   156     0 12269  8041  71
22   7
    356 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
13  0  2 114300  97436   4712  49336  28  50    88    75  112   357  59
22  19
11  0  0 114300  97436   4712  49344   8   0    16     0 12383  9121  58
22  21
    345 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 7  0  0 114300  97436   4716  49344  28  50    88    75  112   358  59
22  19
 7  0  1 114300  97964   4784  49404   0   0    96   188 11222  9017  50
21  29
    332 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 3  0  0 114300  98204   4788  49404  28  50    88    75  112   358  59
22  19
 3  0  0 114300  99344   4792  49432   4   0    40     0 10784  9045  44
18  39
    326 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 5  0  1 114300  99344   4792  49432  28  50    88    75  112   358  59
22  19
 1  0  0 114300  99776   4800  49432  12   0    12    40 9543  7702  43
15  42
    324 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 4  0  0 114300  99776   4800  49432  28  50    88    75  113   358  59
22  19
10  0  0 114300 100636   4816  49608   8   0   200     0 10150  8890  38
14  48
    333 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 2  0  1 114300 100720   4816  49608  28  50    88    75  113   358  59
22  19
12  0  0 114300 101372   4820  49628   0   0    24     0 10896  8939  51
19  30
    331 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 5  0  0 114300 101372   4820  49628  28  50    88    75  113   358  59
22  19
 4  0  0 114300 101100   4868  50096   0   0   516     0 11971  9779  55
21  24
    329 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 1  0  0 114300 101100   4868  50096  28  50    88    75  113   358  59
22  19
 1  0  0 114296 101660   4908  50140   4   0    52   120 9255  7912  43
15  42
    321 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114296 101760   4908  50140  28  50    88    75  113   358  59
22  19
 2  0  0 114296 102880   4924  50144   8   0    16    52 11214  9160  53
17  30
    323 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 2  0  0 114296 102912   4924  50144  28  50    88    75  113   359  59
22  19
 5  0  0 114296 102920   4944  50232   0   0   112     0 12258  9168  60
23  17
    329 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 3  0  0 114296 102920   4944  50232  28  50    88    75  114   359  59
22  19
 2  0  0 114296 102792   4960  50372   0   0   156     0 11192  8830  48
20  32
    337 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 4  0  0 114296 102792   4960  50372  28  50    88    75  114   359  59
22  19
 4  0  0 114296 102792   4960  50404   0   0    32     0 12109  9525  54
21  25
    326 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 1  0  0 114296 102792   4960  50404  28  50    88    75  114   359  59
22  19
 7  0  0 114260 103908   4968  50472   8   0    40    80 11363  9657  49
18  33
    328 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 7  0  0 114260 103908   4968  50472  28  50    88    75  114   359  59
22  19
 0  0  0 114260 104204   4976  50544  28   0   108     0 10663  8804  52
18  30
    325 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260 104204   4976  50544  28  50    88    75  114   359  59
22  19
 2  0  0 114260 104668   4988  50596   0   0    64     0 10596  8753  43
19  39
    326 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 2  0  0 114260 104668   4988  50596  28  50    88    75  115   359  59
22  19
 8  1  1 114260 105468   5016  50652  28   0   120     0 11163  9331  48
16  36
    336 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 8  0  1 114260 105468   5016  50652  28  50    88    75  115   360  59
22  19
 3  0  0 114260 105364   5032  50688   0   0    60     0 12541 10173  55
22  23
    331 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 1  0  0 114260 105364   5032  50688  28  50    88    75  115   360  59
22  19
 7  0  0 114260 105260   5048  50704  12   0    40    16 10463  8565  50
16  34
    326 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 2  0  0 114260 105260   5048  50704  28  50    88    75  115   360  59
22  19
11  0  0 114260 106616   5064  50776   0   0    88     0 12155  8915  59
23  18
    332 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 1  0  0 114260 106616   5064  50776  28  50    88    75  115   360  59
22  19
 5  0  0 114260 106476   5080  50944   0   0   180     0 9364  7202  59
15  25
    327 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 2  0  1 114260 106456   5080  50944  28  50    88    75  115   360  59
22  19
54  0  0 114260 106360   5084  50976   0   0    36     0 3534  2440  14
6  80
    368 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
13  0  0 114260 106556   5084  51008  28  50    88    75  115   360  59
22  19
 0  0  0 114260 106600   5092  51116   4   0   124     0 9835  6055  45
19  37
    385 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260 106600   5092  51116  28  50    88    75  116   360  59
22  19
13  0  0 114260 106848   5108  51280   0   0   172    36 5808  4296  28
10  62
    424 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 6  0  1 114260 106848   5108  51280  28  50    88    75  116   360  59
22  19
 0  0  0 114260 106768   5212  51280  44   0    44   512 4289  3384  23
9  68
    475 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260 106768   5212  51280  28  50    88    75  116   360  59
22  19
 0  0  0 114260 106752   5212  51280   0   0     0     0 2087  1067   4
2  94
    541 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260 106752   5212  51280  28  50    88    75  116   360  59
22  19
 0  0  0 114260 105380   5216  51312   0   0    36     0 1760   732   5
3  92
    613 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260 105276   5216  51312  28  50    88    75  116   360  59
22  19
 0  0  0 114260 100388   5224  51312   0   0     0    16 2126  1326   7
4  89
    625 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260 100212   5224  51312  28  50    88    75  116   360  59
22  19
11  0  0 114260  89200   5228  51308  16   0    20     0 6728  5178  26
13  61
    643 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260  89040   5228  51308  28  50    88    75  116   360  59
22  19
 0  0  0 114260  68260   5228  51316   0   0     8     0 7964  5749  35
19  46
    684 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260  68152   5228  51316  28  50    88    75  116   360  59
22  19
 0  0  0 114260  49980   5276  51348   0   0    28   236 3434  1994  17
12  71
    748 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 1  0  0 114260  49960   5244  51380  28  50    88    75  116   360  59
22  19
 0  0  0 114260  32508   5268  51420   0   0    24   204 3050  1770  15
12  74
    807 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260  32284   5268  51420  28  50    88    75  116   360  59
22  19
 0  1  0 114260  12964   5308  51436   0   0    24   152 3271  1958  16
12  73
    854 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260  12808   5308  51436  28  50    88    75  116   361  59
22  19
 0  0  0 114260  11080   5296  51448   0   0     0     0 1460   611   3
2  94
    853 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 0  0  0 114260  10976   5296  51448  28  50    88    75  116   361  59
22  19
146  0  0 114260   8572   5300  51448   0   0     4     0 1581   415   5
2  92
    842 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
51 224  3 128012   6916   1956  22188  28  50    88    75  118   361  59
22  19
28 223  3 128012   8052   1988  20312  88   0   380     0 12617  7623
72  27   0
    795 connections established
1
   procs                      memory    swap          io     system
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
22 224  3 128012   7876   2012  20548  28  50    88    75  118   362  59
22  19
35 223  3 128012   7368   2036  20688  44   0   208     0 13472  7846
71  29   0
    794 connections established


Mime
View raw message