httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alec Matusis" <matu...@yahoo.com>
Subject RE: [users@httpd] requests time-out under load, no warnings in logs
Date Mon, 01 Oct 2007 22:55:38 GMT
   - how is your iowait ? maybe apache is waiting for disk access.

#vmstat 1

procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
wa

 1  3   2536 698992 195040 211940    0    0     4   187    2     3 14 14 54
17

 1  0   2536 698276 195064 211984    0    0    20     0 8746  9432 27 13 43
17

 0  0   2536 697444 195084 211964    0    0    20     0 8428  8802 26 11 62
1

 0  0   2536 697196 195088 211960    0    0     4     0 7206  6812 19  8 73
0

 1  3   2536 706232 195092 211956    0    0     4  4068 5886  5305 12  7 67
14

 0 82   2536 706300 195092 211956    0    0     0   844 4534  3498  9  4 24
64

 4  0   2536 705516 195092 211956    0    0     0   584 7017  6394 17 10 14
59

 1  0   2536 705460 195092 211956    0    0     0     0 7994  8138 18 10 72
0

 0  0   2536 712564 195100 212016    0    0     8     0 6763  7027 17  8 74
0

 0  0   2536 712692 195112 212004    0    0    12     0 6201  8271 15  7 77
1

 1  1   2536 712256 195116 212000    0    0    20  2372 6276  5845 14  6 78
2

 1 32   2536 712436 195128 211988    0    0    12  1584 6633  7025 15  7  9
69

 2  4   2536 722980 195128 211988    0    0     0   592 7114  7063 16 10 10
64

 1  0   2536 722796 195140 212044    0    0    32     0 7215  7184 17  9 71
4

 5  0   2536 722300 195164 212088    0    0    24   168 8066  9076 25 11 61
2

 1  0   2536 722180 195168 212084    0    0     4     0 7529  7952 19 10 70
0

 2 27   2536 720712 195168 211268    0    0     0  3792 6718  6542 20 11 28
41

 4  7   2536 730984 195168 211268    0    0     0  1616 10398 10589 44 22  0
35

 3  5   2536 731864 195172 211264    0    0     4   740 8842 10712 27 14  4
54

 1  2   2536 732116 195184 211320    0    0    12    40 7049  7312 17  9 43
32

 

This is when apache is running normally. Iowait does not seem to be a
problem, but I did not check during the crashes.

 

   - are your sure that you don't have a network bottleneck ? load balancer,
firewall, network interface,..

 

We have a Twisted server running on the same machine, and it works OK during
the crashes- so it does not seem to be a network bottleneck. Plus, system
log shows only this during crashes, which indicates apache-specific
bottleneck:

Oct  1 13:21:56 web10 kernel: possible SYN flooding on port 80. Sending
cookies.

Oct  1 13:22:56 web10 kernel: possible SYN flooding on port 80. Sending
cookies.

Oct  1 13:23:57 web10 kernel: possible SYN flooding on port 80. Sending
cookies. 

 

  - do you serve static files only with you apache ? if you use php,
cgi,mod_jk,mysql,... You should have a look to them maybe they are hanging
up.



We are using mod_python. I posted to mod_python list as well.

 

  - if you use thread safe modules (not php) and if you have to serve a lot
of little requests, maybe you can consider to use mpm worker instead of
prefork.

 

This is a good idea. We will try it. Does it give significant performance
bonus? Our average request size is only 1-2K.

 

Alec

 

 

From: Bj [mailto:bjkhanz@gmail.com] 
Sent: Monday, October 01, 2007 3:15 AM
To: users@httpd.apache.org
Subject: Re: [users@httpd] requests time-out under load, no warnings in logs

 


Hi,

When your server is slow: 
   - how is your iowait ? maybe apache is waiting for disk access. 
   - are your sure that you don't have a network bottleneck ? load balancer,
firewall, network interface,.. 
  - do you serve static files only with you apache ? if you use php,
cgi,mod_jk,mysql,... You should have a look to them maybe they are hanging
up.
  - if you use thread safe modules (not php) and if you have to serve a lot
of little requests, maybe you can consider to use mpm worker instead of
prefork. 

-- 
Bj



On 10/1/07, Alec Matusis <matusis@yahoo.com> wrote:

Hi Christian,

> How does your apache look memory-wise in top/htop?

When Apache is "slow", it has about 200 processes with 11-15MB RSS each,
soat least 1GB of physical RAM should be unused.
The load average drops from normal 16.0 (this is a 4 CPU machine) to only
6.0 when apache becomes "slow".


It would be interesting to see an example of a reasonable choice for
MaxMemFree, after 2 days of searching in Google I cannot find any guidelines
for setting it. 



> -----Original Message-----
> From: Christian Folini [mailto:christian.folini@post.ch]
> Sent: Sunday, September 30, 2007 11:55 PM
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] requests time-out under load, no warnings in
> logs
>
> Hey Alec,
>
> This looks like a tricky problem. Have you looked at the traffic 
> using tcpdump or ethereal when the machine hangs? You might
> also try out mod_forensic or mod_security and add mod_security
> internal timestamps to the access-log. Using these logs you will
> get a clearer idea where it hangs. 
>
> How does your apache look memory-wise in top/htop?
> Are you sure it is not an application issue - even if it shows
> serving static files. A reverse proxy setup for testing purposes
> could seperate application and static file serving to 
> give you a clearer idea in this regard.
>
> These are just a lot of ideas. I do not really know how to
> solve your problem, but this would give you a clearer idea
> of the problem.
> 
> I am not familiar with best practice values for MaxMemFree.
>
> regs,
>
> Christian
>
>
>
> On Sun, Sep 30, 2007 at 03:54:59PM -0700, Alec Matusis wrote:
> > We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that
> suddenly
> > becomes very slow- requests either time out, or it takes 10s to serve
> a 1K
> > thumbnail.
> > It is usually correlated with load spikes, but not perfectly (by 
> looking at
> > the bandwidth graph)
> > Then it requires a stop/start, and it becomes fast again.
> > This has been happening before, and it was always accompanied with
> >
> > [error] server reached MaxClients setting, consider raising the
> MaxClients
> > setting
> >
> > in the apache log.
> >
> > We raised MaxClients    from 200 to 300, and now it happens with no 
> warning
> > in any logs at all, except with the usual
> >
> > web10 kernel: possible SYN flooding on port 80. Sending cookies. in
> > /var/log/messages (this has always happened when apache hung from 
> overload).
> >
> > What worries me is that there are no log messages in apache logs at
> all when
> > this happens.
> >
> > This is the relevant part of httpd.conf 
> >
> > ServerLimit 300
> > # we are using prefork MPM
> > <IfModule prefork.c>
> > StartServers          10
> > MinSpareServers       5
> > MaxSpareServers       20 
> > MaxClients            300
> > MaxRequestsPerChild   10000
> > MaxMemFree            2560
> > </IfModule>
> >
> > The server has 4GB of RAM and 4GB of swap. When apache "crashes" i.e.
> > becomes very slow, the swap size is still 0 and vmstat shows no
> swapping.
> > I suspect the problem may be in
> >
> > MaxMemFree   2560
> >
> > but then I would expect some kind of "out of memory" errors in the 
> logs?
> > Another question: what is the good choice of MaxMemFree value? We
> want to
> > leave 512MB-1GB or RAM on this machine for other tasks.
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL: http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org 
> >    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project. 
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org 
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL: <http://httpd.apache.org/userslist.html>
http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

 


Mime
View raw message